Jul 28 17:07:23.604235 ip-10-0-138-231 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 28 17:07:23.604247 ip-10-0-138-231 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 28 17:07:23.604256 ip-10-0-138-231 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 28 17:07:23.604561 ip-10-0-138-231 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 28 17:07:33.682139 ip-10-0-138-231 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 28 17:07:33.682158 ip-10-0-138-231 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot bd95527cbd7a4378ba47d3f99124f74b --
Jul 28 17:09:10.079948 ip-10-0-138-231 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 17:09:10.528274 ip-10-0-138-231 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:09:10.528274 ip-10-0-138-231 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 17:09:10.528274 ip-10-0-138-231 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:09:10.528274 ip-10-0-138-231 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 17:09:10.528274 ip-10-0-138-231 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:09:10.531085 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.531003    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 17:09:10.536059 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536044    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:09:10.536059 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536059    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:09:10.536121 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536065    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:09:10.536121 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536070    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:09:10.536121 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536074    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:09:10.536121 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536077    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:09:10.536121 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536079    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:09:10.536121 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536082    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:09:10.536121 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536085    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:09:10.536121 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536088    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:09:10.536121 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536090    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:09:10.536121 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536093    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:09:10.536121 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536096    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:09:10.536121 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536099    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:09:10.536121 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536102    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:09:10.536121 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536104    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:09:10.536121 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536107    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:09:10.536121 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536110    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:09:10.536121 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536113    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:09:10.536121 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536115    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:09:10.536121 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536118    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:09:10.536572 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536121    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:09:10.536572 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536123    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:09:10.536572 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536126    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:09:10.536572 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536130    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:09:10.536572 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536133    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:09:10.536572 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536136    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:09:10.536572 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536140    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:09:10.536572 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536143    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:09:10.536572 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536145    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:09:10.536572 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536150    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:09:10.536572 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536153    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:09:10.536572 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536156    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:09:10.536572 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536159    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:09:10.536572 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536162    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:09:10.536572 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536165    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:09:10.536572 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536167    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:09:10.536572 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536170    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:09:10.536572 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536173    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:09:10.536572 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536176    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:09:10.536572 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536178    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:09:10.537070 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536181    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:09:10.537070 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536183    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:09:10.537070 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536186    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:09:10.537070 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536188    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:09:10.537070 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536191    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:09:10.537070 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536194    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:09:10.537070 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536196    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:09:10.537070 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536199    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:09:10.537070 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536202    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:09:10.537070 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536204    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:09:10.537070 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536207    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:09:10.537070 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536209    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:09:10.537070 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536212    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:09:10.537070 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536215    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:09:10.537070 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536218    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:09:10.537070 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536221    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:09:10.537070 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536223    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:09:10.537070 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536226    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:09:10.537070 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536228    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:09:10.537529 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536231    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:09:10.537529 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536233    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:09:10.537529 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536236    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:09:10.537529 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536239    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:09:10.537529 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536241    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:09:10.537529 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536244    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:09:10.537529 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536246    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:09:10.537529 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536249    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:09:10.537529 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536252    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:09:10.537529 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536255    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:09:10.537529 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536257    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:09:10.537529 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536261    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:09:10.537529 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536264    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:09:10.537529 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536266    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:09:10.537529 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536269    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:09:10.537529 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536271    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:09:10.537529 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536274    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:09:10.537529 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536276    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:09:10.537529 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536280    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:09:10.537529 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536283    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:09:10.538075 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536285    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:09:10.538075 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536288    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:09:10.538075 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536290    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:09:10.538075 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536293    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:09:10.538075 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536295    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:09:10.538075 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536298    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:09:10.538075 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536301    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:09:10.538075 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536304    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:09:10.538075 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536308    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:09:10.538075 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536312    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:09:10.538075 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536315    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:09:10.538075 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536318    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:09:10.538075 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536320    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:09:10.538075 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536323    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:09:10.538075 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536326    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:09:10.538075 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536328    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:09:10.538075 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536331    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:09:10.538075 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536334    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:09:10.538075 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.536336    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:09:10.538075 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537047    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:09:10.538561 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537053    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:09:10.538561 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537056    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:09:10.538561 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537059    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:09:10.538561 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537063    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:09:10.538561 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537070    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:09:10.538561 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537073    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:09:10.538561 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537076    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:09:10.538561 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537079    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:09:10.538561 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537082    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:09:10.538561 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537084    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:09:10.538561 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537087    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:09:10.538561 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537090    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:09:10.538561 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537092    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:09:10.538561 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537095    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:09:10.538561 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537097    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:09:10.538561 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537100    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:09:10.538561 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537102    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:09:10.538561 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537105    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:09:10.538561 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537108    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:09:10.538561 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537111    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:09:10.539074 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537113    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:09:10.539074 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537116    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:09:10.539074 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537119    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:09:10.539074 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537122    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:09:10.539074 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537125    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:09:10.539074 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537127    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:09:10.539074 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537131    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:09:10.539074 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537133    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:09:10.539074 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537136    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:09:10.539074 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537138    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:09:10.539074 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537141    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:09:10.539074 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537144    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:09:10.539074 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537149    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:09:10.539074 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537153    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:09:10.539074 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537156    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:09:10.539074 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537158    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:09:10.539074 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537161    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:09:10.539074 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537164    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:09:10.539074 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537168    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:09:10.539547 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537170    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:09:10.539547 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537174    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:09:10.539547 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537176    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:09:10.539547 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537181    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:09:10.539547 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537183    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:09:10.539547 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537185    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:09:10.539547 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537188    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:09:10.539547 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537191    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:09:10.539547 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537194    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:09:10.539547 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537196    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:09:10.539547 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537199    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:09:10.539547 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537201    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:09:10.539547 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537204    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:09:10.539547 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537207    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:09:10.539547 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537209    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:09:10.539547 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537212    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:09:10.539547 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537214    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:09:10.539547 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537217    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:09:10.539547 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537219    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:09:10.540042 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537222    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:09:10.540042 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537225    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:09:10.540042 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537227    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:09:10.540042 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537230    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:09:10.540042 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537233    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:09:10.540042 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537236    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:09:10.540042 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537238    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:09:10.540042 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537240    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:09:10.540042 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537243    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:09:10.540042 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537246    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:09:10.540042 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537248    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:09:10.540042 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537250    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:09:10.540042 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537253    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:09:10.540042 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537256    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:09:10.540042 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537258    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:09:10.540042 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537261    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:09:10.540042 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537263    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:09:10.540042 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537265    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:09:10.540042 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537268    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:09:10.540042 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537271    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:09:10.540532 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537273    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:09:10.540532 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537276    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:09:10.540532 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537278    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:09:10.540532 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537283    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:09:10.540532 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537287    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:09:10.540532 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537290    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:09:10.540532 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537293    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:09:10.540532 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537296    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:09:10.540532 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537299    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:09:10.540532 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537302    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:09:10.540532 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537305    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:09:10.540532 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537308    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:09:10.540532 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537311    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:09:10.540532 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537313    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:09:10.540532 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537316    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:09:10.540532 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537319    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:09:10.540532 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537321    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:09:10.540532 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537324    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:09:10.540532 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537326    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:09:10.540532 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537329    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:09:10.541069 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537724    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:09:10.541069 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537731    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:09:10.541069 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537734    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:09:10.541069 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537737    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:09:10.541069 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537740    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:09:10.541069 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537743    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:09:10.541069 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537745    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:09:10.541069 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537748    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:09:10.541069 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537750    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:09:10.541069 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537753    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:09:10.541069 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537756    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:09:10.541069 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537758    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:09:10.541069 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537761    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:09:10.541069 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537763    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:09:10.541069 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537766    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:09:10.541069 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537769    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:09:10.541069 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537771    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:09:10.541069 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537774    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:09:10.541069 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537777    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:09:10.541069 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537780    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:09:10.541549 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537782    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:09:10.541549 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537785    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:09:10.541549 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537788    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:09:10.541549 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537791    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:09:10.541549 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537793    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:09:10.541549 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537796    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:09:10.541549 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537799    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:09:10.541549 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537801    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:09:10.541549 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537804    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:09:10.541549 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537807    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:09:10.541549 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537809    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:09:10.541549 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537812    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:09:10.541549 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537814    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:09:10.541549 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537816    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:09:10.541549 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537819    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:09:10.541549 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537821    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:09:10.541549 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537824    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:09:10.541549 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537827    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:09:10.541549 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537829    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:09:10.541549 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537831    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:09:10.542044 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537834    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:09:10.542044 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537837    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:09:10.542044 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537839    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:09:10.542044 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537842    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:09:10.542044 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537844    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:09:10.542044 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537847    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:09:10.542044 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537849    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:09:10.542044 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537852    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:09:10.542044 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537858    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:09:10.542044 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537860    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:09:10.542044 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537863    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:09:10.542044 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537868    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:09:10.542044 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537872    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:09:10.542044 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537874    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:09:10.542044 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537877    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:09:10.542044 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537880    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:09:10.542044 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537883    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:09:10.542044 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537886    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:09:10.542044 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537888    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:09:10.542044 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537891    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:09:10.542524 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537893    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:09:10.542524 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537897    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:09:10.542524 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537901    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:09:10.542524 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537905    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:09:10.542524 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537908    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:09:10.542524 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537912    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:09:10.542524 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537914    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:09:10.542524 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537917    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:09:10.542524 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537920    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:09:10.542524 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537922    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:09:10.542524 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537925    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:09:10.542524 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537928    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:09:10.542524 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537931    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:09:10.542524 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537933    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:09:10.542524 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537935    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:09:10.542524 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537938    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:09:10.542524 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537940    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:09:10.542524 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537943    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:09:10.542524 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537946    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:09:10.543033 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537949    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:09:10.543033 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537953    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:09:10.543033 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537956    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:09:10.543033 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537959    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:09:10.543033 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537961    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:09:10.543033 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537964    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:09:10.543033 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537966    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:09:10.543033 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537969    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:09:10.543033 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537971    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:09:10.543033 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537974    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:09:10.543033 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537976    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:09:10.543033 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537979    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:09:10.543033 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537981    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:09:10.543033 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537984    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:09:10.543033 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537987    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:09:10.543033 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537989    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:09:10.543033 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537992    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:09:10.543033 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537995    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:09:10.543033 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.537997    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:09:10.543489 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538000    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:09:10.543489 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538060    2581 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 17:09:10.543489 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538071    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 17:09:10.543489 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538083    2581 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 17:09:10.543489 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538089    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 17:09:10.543489 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538093    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 17:09:10.543489 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538097    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 17:09:10.543489 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538101    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 17:09:10.543489 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538105    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 17:09:10.543489 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538108    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 17:09:10.543489 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538110    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 17:09:10.543489 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538114    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 17:09:10.543489 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538117    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 17:09:10.543489 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538121    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 17:09:10.543489 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538124    2581 flags.go:64] FLAG: --cgroup-root=""
Jul 28 17:09:10.543489 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538127    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 17:09:10.543489 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538130    2581 flags.go:64] FLAG: --client-ca-file=""
Jul 28 17:09:10.543489 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538133    2581 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 17:09:10.543489 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538136    2581 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 17:09:10.543489 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538143    2581 flags.go:64] FLAG: --cluster-domain=""
Jul 28 17:09:10.543489 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538145    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 17:09:10.543489 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538149    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 17:09:10.543489 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538152    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 17:09:10.543489 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538155    2581 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 17:09:10.544084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538158    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 17:09:10.544084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538161    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 17:09:10.544084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538164    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 17:09:10.544084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538169    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 17:09:10.544084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538172    2581 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 17:09:10.544084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538175    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 17:09:10.544084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538177    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 17:09:10.544084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538181    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 17:09:10.544084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538183    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 17:09:10.544084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538187    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 17:09:10.544084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538190    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 17:09:10.544084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538193    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 17:09:10.544084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538195    2581 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 17:09:10.544084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538198    2581 flags.go:64] FLAG: --enable-server="true"
Jul 28 17:09:10.544084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538201    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 17:09:10.544084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538206    2581 flags.go:64] FLAG: --event-burst="100"
Jul 28 17:09:10.544084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538209    2581 flags.go:64] FLAG: --event-qps="50"
Jul 28 17:09:10.544084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538212    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 17:09:10.544084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538215    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 17:09:10.544084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538218    2581 flags.go:64] FLAG: --eviction-hard=""
Jul 28 17:09:10.544084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538221    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 17:09:10.544084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538224    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 17:09:10.544084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538227    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 17:09:10.544084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538230    2581 flags.go:64] FLAG: --eviction-soft=""
Jul 28 17:09:10.544084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538233    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 17:09:10.544703 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538236    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 17:09:10.544703 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538239    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 17:09:10.544703 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538242    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 17:09:10.544703 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538244    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 17:09:10.544703 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538247    2581 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 17:09:10.544703 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538250    2581 flags.go:64] FLAG: --feature-gates=""
Jul 28 17:09:10.544703 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538254    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 17:09:10.544703 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538257    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 17:09:10.544703 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538260    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 17:09:10.544703 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538263    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 17:09:10.544703 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538267    2581 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 17:09:10.544703 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538270    2581 flags.go:64] FLAG: --help="false"
Jul 28 17:09:10.544703 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538272    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-138-231.ec2.internal"
Jul 28 17:09:10.544703 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538275    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 17:09:10.544703 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538278    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 17:09:10.544703 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538281    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 17:09:10.544703 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538284    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 17:09:10.544703 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538288    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 17:09:10.544703 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538291    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 17:09:10.544703 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538293    2581 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 17:09:10.544703 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538296    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 17:09:10.544703 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538299    2581 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 17:09:10.544703 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538302    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 17:09:10.545283 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538305    2581 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 17:09:10.545283 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538308    2581 flags.go:64] FLAG: --kube-reserved=""
Jul 28 17:09:10.545283 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538311    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 17:09:10.545283 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538314    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 17:09:10.545283 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538317    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 17:09:10.545283 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538320    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 17:09:10.545283 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538324    2581 flags.go:64] FLAG: --lock-file=""
Jul 28 17:09:10.545283 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538327    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 17:09:10.545283 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538330    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 17:09:10.545283 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538332    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 17:09:10.545283 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538337    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 17:09:10.545283 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538340    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 17:09:10.545283 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538343    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 17:09:10.545283 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538346    2581 flags.go:64] FLAG: --logging-format="text"
Jul 28 17:09:10.545283 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538349    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 17:09:10.545283 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538352    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 17:09:10.545283 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538354    2581 flags.go:64] FLAG: --manifest-url=""
Jul 28 17:09:10.545283 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538357    2581 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 17:09:10.545283 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538361    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 17:09:10.545283 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538365    2581 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 17:09:10.545283 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538369    2581 flags.go:64] FLAG: --max-pods="110"
Jul 28 17:09:10.545283 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538372    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 17:09:10.545283 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538375    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 17:09:10.545283 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538378    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 17:09:10.545283 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538381    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 17:09:10.545901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538384    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 17:09:10.545901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538388    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 17:09:10.545901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538391    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 17:09:10.545901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538398    2581 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 17:09:10.545901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538401    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 17:09:10.545901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538404    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 17:09:10.545901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538406    2581 flags.go:64] FLAG: --pod-cidr=""
Jul 28 17:09:10.545901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538410    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 17:09:10.545901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538415    2581 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 17:09:10.545901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538418    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 17:09:10.545901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538421    2581 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 17:09:10.545901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538424    2581 flags.go:64] FLAG: --port="10250"
Jul 28 17:09:10.545901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538427    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 17:09:10.545901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538430    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00e7b117e146c39de"
Jul 28 17:09:10.545901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538433    2581 flags.go:64] FLAG: --qos-reserved=""
Jul 28 17:09:10.545901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538436    2581 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 17:09:10.545901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538438    2581 flags.go:64] FLAG: --register-node="true"
Jul 28 17:09:10.545901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538441    2581 flags.go:64] FLAG: --register-with-taints=""
Jul 28 17:09:10.545901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538444    2581 flags.go:64] FLAG: --registry-burst="10"
Jul 28 17:09:10.545901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538447    2581 flags.go:64] FLAG: --registry-qps="5"
Jul 28 17:09:10.545901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538450    2581 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 17:09:10.545901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538452    2581 flags.go:64] FLAG: --reserved-memory=""
Jul 28 17:09:10.545901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538456    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 17:09:10.545901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538459    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 17:09:10.545901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538461    2581 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 17:09:10.546540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538464    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 17:09:10.546540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538468    2581 flags.go:64] FLAG: --runonce="false"
Jul 28 17:09:10.546540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538471    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 17:09:10.546540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538474    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 17:09:10.546540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538477    2581 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 17:09:10.546540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538479    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 17:09:10.546540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538482    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 17:09:10.546540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538485    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 17:09:10.546540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538488    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 17:09:10.546540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538491    2581 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 17:09:10.546540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538494    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 17:09:10.546540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538497    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 17:09:10.546540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538499    2581 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 17:09:10.546540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538502    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 17:09:10.546540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538505    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 17:09:10.546540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538508    2581 flags.go:64] FLAG: --system-cgroups=""
Jul 28 17:09:10.546540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538511    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 17:09:10.546540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538516    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 17:09:10.546540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538519    2581 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 17:09:10.546540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538521    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 17:09:10.546540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538526    2581 flags.go:64] FLAG: --tls-min-version=""
Jul 28 17:09:10.546540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538530    2581 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 17:09:10.546540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538533    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 17:09:10.546540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538536    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 17:09:10.546540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538538    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 17:09:10.547232 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538541    2581 flags.go:64] FLAG: --v="2"
Jul 28 17:09:10.547232 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538545    2581 flags.go:64] FLAG: --version="false"
Jul 28 17:09:10.547232 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538549    2581 flags.go:64] FLAG: --vmodule=""
Jul 28 17:09:10.547232 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538552    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 17:09:10.547232 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.538556    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 17:09:10.547232 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538683    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:09:10.547232 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538687    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:09:10.547232 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538691    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:09:10.547232 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538695    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:09:10.547232 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538698    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:09:10.547232 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538701    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:09:10.547232 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538704    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:09:10.547232 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538707    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:09:10.547232 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538711    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:09:10.547232 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538713    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:09:10.547232 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538716    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:09:10.547232 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538719    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:09:10.547232 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538721    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:09:10.547232 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538724    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:09:10.547232 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538727    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:09:10.547735 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538732    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:09:10.547735 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538735    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:09:10.547735 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538738    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:09:10.547735 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538742    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:09:10.547735 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538744    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:09:10.547735 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538747    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:09:10.547735 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538750    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:09:10.547735 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538753    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:09:10.547735 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538757    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:09:10.547735 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538760    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:09:10.547735 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538762    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:09:10.547735 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538765    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:09:10.547735 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538767    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:09:10.547735 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538770    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:09:10.547735 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538773    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:09:10.547735 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538776    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:09:10.547735 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538778    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:09:10.547735 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538781    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:09:10.547735 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538784    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:09:10.547735 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538786    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:09:10.548247 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538790    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:09:10.548247 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538793    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:09:10.548247 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538796    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:09:10.548247 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538798    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:09:10.548247 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538801    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:09:10.548247 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538804    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:09:10.548247 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538806    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:09:10.548247 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538809    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:09:10.548247 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538811    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:09:10.548247 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538813    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:09:10.548247 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538816    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:09:10.548247 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538819    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:09:10.548247 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538821    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:09:10.548247 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538824    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:09:10.548247 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538826    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:09:10.548247 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538829    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:09:10.548247 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538832    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:09:10.548247 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538834    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:09:10.548247 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538837    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:09:10.548731 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538839    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:09:10.548731 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538843    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:09:10.548731 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538846    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:09:10.548731 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538848    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:09:10.548731 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538851    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:09:10.548731 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538853    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:09:10.548731 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538856    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:09:10.548731 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538858    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:09:10.548731 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538861    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:09:10.548731 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538863    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:09:10.548731 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538866    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:09:10.548731 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538869    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:09:10.548731 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538871    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:09:10.548731 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538875    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:09:10.548731 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538878    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:09:10.548731 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538880    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:09:10.548731 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538883    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:09:10.548731 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538885    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:09:10.548731 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538888    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:09:10.548731 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538890    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:09:10.549224 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538893    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:09:10.549224 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538895    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:09:10.549224 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538898    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:09:10.549224 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538901    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:09:10.549224 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538903    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:09:10.549224 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538906    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:09:10.549224 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538909    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:09:10.549224 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538911    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:09:10.549224 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538914    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:09:10.549224 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538916    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:09:10.549224 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538919    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:09:10.549224 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538922    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:09:10.549224 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538924    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:09:10.549224 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538928    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:09:10.549224 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538931    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:09:10.549224 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538933    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:09:10.549224 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538936    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:09:10.549224 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538938    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:09:10.549224 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538941    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:09:10.549224 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538943    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:09:10.549722 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538947    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:09:10.549722 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538950    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:09:10.549722 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538953    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:09:10.549722 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538956    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:09:10.549722 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.538959    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:09:10.549722 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.539715    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:09:10.549722 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.546710    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 17:09:10.549722 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.546725    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 17:09:10.549722 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.546743    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 17:09:10.549722 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.546749    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 17:09:10.549722 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546804    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:09:10.549722 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546809    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:09:10.549722 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546812    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:09:10.549722 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546815    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:09:10.550072 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546818    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:09:10.550072 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546821    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:09:10.550072 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546824    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:09:10.550072 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546827    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:09:10.550072 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546829    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:09:10.550072 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546832    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:09:10.550072 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546835    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:09:10.550072 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546838    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:09:10.550072 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546840    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:09:10.550072 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546843    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:09:10.550072 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546846    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:09:10.550072 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546848    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:09:10.550072 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546851    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:09:10.550072 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546854    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:09:10.550072 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546856    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:09:10.550072 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546859    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:09:10.550072 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546861    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:09:10.550072 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546864    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:09:10.550072 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546867    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:09:10.550072 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546869    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:09:10.550548 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546872    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:09:10.550548 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546874    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:09:10.550548 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546877    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:09:10.550548 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546879    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:09:10.550548 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546882    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:09:10.550548 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546886    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:09:10.550548 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546889    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:09:10.550548 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546891    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:09:10.550548 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546894    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:09:10.550548 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546897    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:09:10.550548 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546899    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:09:10.550548 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546902    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:09:10.550548 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546904    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:09:10.550548 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546908    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:09:10.550548 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546912    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:09:10.550548 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546915    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:09:10.550548 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546918    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:09:10.550548 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546921    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:09:10.550548 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546925    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:09:10.551017 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546928    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:09:10.551017 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546930    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:09:10.551017 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546933    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:09:10.551017 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546935    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:09:10.551017 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546938    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:09:10.551017 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546941    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:09:10.551017 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546943    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:09:10.551017 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546945    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:09:10.551017 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546949    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:09:10.551017 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546951    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:09:10.551017 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546954    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:09:10.551017 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546956    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:09:10.551017 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546958    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:09:10.551017 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546961    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:09:10.551017 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546964    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:09:10.551017 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546966    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:09:10.551017 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546969    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:09:10.551017 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546972    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:09:10.551017 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546975    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:09:10.551017 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546979    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:09:10.551513 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546981    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:09:10.551513 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546984    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:09:10.551513 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546986    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:09:10.551513 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546989    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:09:10.551513 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546992    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:09:10.551513 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.546997    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:09:10.551513 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547000    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:09:10.551513 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547003    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:09:10.551513 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547006    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:09:10.551513 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547009    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:09:10.551513 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547011    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:09:10.551513 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547014    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:09:10.551513 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547017    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:09:10.551513 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547019    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:09:10.551513 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547022    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:09:10.551513 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547024    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:09:10.551513 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547027    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:09:10.551513 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547029    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:09:10.551513 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547032    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:09:10.551990 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547034    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:09:10.551990 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547037    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:09:10.551990 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547040    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:09:10.551990 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547042    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:09:10.551990 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547045    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:09:10.551990 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547047    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:09:10.551990 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547050    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:09:10.551990 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547053    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:09:10.551990 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547055    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:09:10.551990 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547058    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:09:10.551990 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547061    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:09:10.551990 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547063    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:09:10.551990 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547066    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:09:10.551990 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547070    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:09:10.551990 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547072    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:09:10.551990 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547075    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:09:10.551990 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547077    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:09:10.552404 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.547082    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:09:10.552404 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547192    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:09:10.552404 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547196    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:09:10.552404 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547199    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:09:10.552404 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547202    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:09:10.552404 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547205    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:09:10.552404 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547208    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:09:10.552404 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547211    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:09:10.552404 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547213    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:09:10.552404 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547216    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:09:10.552404 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547219    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:09:10.552404 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547221    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:09:10.552404 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547224    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:09:10.552404 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547227    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:09:10.552404 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547229    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:09:10.552792 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547232    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:09:10.552792 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547236    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:09:10.552792 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547239    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:09:10.552792 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547242    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:09:10.552792 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547244    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:09:10.552792 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547247    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:09:10.552792 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547250    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:09:10.552792 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547252    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:09:10.552792 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547256    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:09:10.552792 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547259    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:09:10.552792 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547261    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:09:10.552792 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547264    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:09:10.552792 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547266    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:09:10.552792 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547269    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:09:10.552792 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547273    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:09:10.552792 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547276    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:09:10.552792 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547279    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:09:10.552792 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547281    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:09:10.552792 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547284    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:09:10.552792 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547287    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:09:10.553280 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547289    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:09:10.553280 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547292    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:09:10.553280 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547294    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:09:10.553280 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547297    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:09:10.553280 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547299    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:09:10.553280 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547302    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:09:10.553280 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547304    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:09:10.553280 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547307    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:09:10.553280 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547310    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:09:10.553280 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547312    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:09:10.553280 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547315    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:09:10.553280 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547317    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:09:10.553280 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547320    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:09:10.553280 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547322    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:09:10.553280 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547325    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:09:10.553280 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547328    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:09:10.553280 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547330    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:09:10.553280 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547333    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:09:10.553280 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547335    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:09:10.553280 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547338    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:09:10.553822 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547341    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:09:10.553822 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547345    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:09:10.553822 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547348    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:09:10.553822 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547351    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:09:10.553822 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547355    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:09:10.553822 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547358    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:09:10.553822 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547361    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:09:10.553822 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547364    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:09:10.553822 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547367    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:09:10.553822 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547370    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:09:10.553822 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547373    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:09:10.553822 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547376    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:09:10.553822 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547378    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:09:10.553822 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547381    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:09:10.553822 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547383    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:09:10.553822 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547386    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:09:10.553822 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547388    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:09:10.553822 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547390    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:09:10.553822 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547393    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:09:10.554281 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547395    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:09:10.554281 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547399    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:09:10.554281 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547402    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:09:10.554281 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547404    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:09:10.554281 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547407    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:09:10.554281 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547409    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:09:10.554281 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547412    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:09:10.554281 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547415    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:09:10.554281 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547417    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:09:10.554281 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547420    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:09:10.554281 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547422    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:09:10.554281 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547425    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:09:10.554281 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547427    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:09:10.554281 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547445    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:09:10.554281 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547448    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:09:10.554281 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547451    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:09:10.554281 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547454    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:09:10.554281 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547457    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:09:10.554281 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547460    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:09:10.554755 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547463    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:09:10.554755 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547466    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:09:10.554755 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547469    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:09:10.554755 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547472    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:09:10.554755 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547474    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:09:10.554755 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547477    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:09:10.554755 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:10.547479    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:09:10.554755 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.547484    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:09:10.554755 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.547577    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 17:09:10.554755 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.551734    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 17:09:10.554755 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.553476    2581 server.go:1019] "Starting client certificate rotation"
Jul 28 17:09:10.554755 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.553572    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:09:10.554755 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.553646    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:09:10.556247 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.556235    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:09:10.562386 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.562365    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:09:10.572468 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.572449    2581 log.go:25] "Validated CRI v1 runtime API"
Jul 28 17:09:10.575255 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.575241    2581 log.go:25] "Validated CRI v1 image API"
Jul 28 17:09:10.576130 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.576115    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 17:09:10.578723 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.578703    2581 fs.go:135] Filesystem UUIDs: map[3c60f312-1a2f-4d86-a972-a41f2cfa2a82:/dev/nvme0n1p4 55d29c35-ac7d-4a1a-80a7-c05d39d351b4:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 28 17:09:10.578807 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.578721    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 17:09:10.582982 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.582867    2581 manager.go:216] Machine: {Timestamp:2026-07-28 17:09:10.582394711 +0000 UTC m=+0.373340885 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103629 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20946dbef5d5132c2792706c5d7383 SystemUUID:ec20946d-bef5-d513-2c27-92706c5d7383 BootID:bd95527c-bd7a-4378-ba47-d3f99124f74b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:07:01:4c:51:1f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:07:01:4c:51:1f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:19:53:e0:4b:cd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 17:09:10.582982 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.582968    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 17:09:10.583133 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.583058    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 17:09:10.584181 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.584162    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:09:10.586331 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.586305    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 17:09:10.586452 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.586334    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-231.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 17:09:10.587217 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.587207    2581 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 17:09:10.587249 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.587218    2581 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 17:09:10.587249 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.587226    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:09:10.588088 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.588074    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:09:10.589356 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.589346    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 17:09:10.592173 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.592163    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:09:10.592292 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.592283    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 17:09:10.592338 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.592313    2581 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 17:09:10.592338 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.592326    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 17:09:10.592395 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.592348    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 17:09:10.592395 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.592360    2581 kubelet.go:398] "Adding apiserver pod source"
Jul 28 17:09:10.592395 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.592371    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 17:09:10.592691 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.592680    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:09:10.592737 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.592698    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:09:10.595014 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.595000    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 17:09:10.597378 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.597366    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 17:09:10.597421 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.597395    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 17:09:10.600073 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.600059    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 17:09:10.600134 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.600079    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 17:09:10.600134 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.600093    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 17:09:10.600134 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.600100    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 17:09:10.600134 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.600108    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 17:09:10.600134 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.600114    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 17:09:10.600134 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.600122    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 17:09:10.600134 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.600127    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 17:09:10.600134 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.600134    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 17:09:10.600333 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.600140    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 17:09:10.600333 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.600153    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 17:09:10.600333 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.600158    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 17:09:10.601098 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.601088    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 17:09:10.601098 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.601098    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 17:09:10.603972 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.603958    2581 server.go:1268] "Started kubelet"
Jul 28 17:09:10.604098 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.604037    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 17:09:10.607829 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:10.607455    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-231.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 17:09:10.607829 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.604052    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 17:09:10.607829 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.607547    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 17:09:10.607829 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.607553    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-231.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 17:09:10.607764 ip-10-0-138-231 systemd[1]: Started Kubernetes Kubelet.
Jul 28 17:09:10.608202 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:10.607915    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 17:09:10.609796 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.609778    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 17:09:10.612966 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.612859    2581 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 17:09:10.615061 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:10.615041    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 17:09:10.616120 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:10.614928    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-231.ec2.internal.18c683316cf9df21  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-231.ec2.internal,UID:ip-10-0-138-231.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-231.ec2.internal,},FirstTimestamp:2026-07-28 17:09:10.603931425 +0000 UTC m=+0.394877598,LastTimestamp:2026-07-28 17:09:10.603931425 +0000 UTC m=+0.394877598,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-231.ec2.internal,}"
Jul 28 17:09:10.621409 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.621394    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 17:09:10.621409 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.621403    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 17:09:10.621535 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:10.621509    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-231.ec2.internal\" not found"
Jul 28 17:09:10.622226 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.622197    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 17:09:10.622226 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.622200    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 17:09:10.622226 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.622223    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 17:09:10.622395 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.622320    2581 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 17:09:10.622395 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.622328    2581 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 17:09:10.622395 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.622366    2581 factory.go:153] Registering CRI-O factory
Jul 28 17:09:10.622395 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.622380    2581 factory.go:223] Registration of the crio container factory successfully
Jul 28 17:09:10.622576 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.622445    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 17:09:10.622576 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.622456    2581 factory.go:55] Registering systemd factory
Jul 28 17:09:10.622576 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.622462    2581 factory.go:223] Registration of the systemd container factory successfully
Jul 28 17:09:10.622576 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.622480    2581 factory.go:103] Registering Raw factory
Jul 28 17:09:10.622576 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.622493    2581 manager.go:1196] Started watching for new ooms in manager
Jul 28 17:09:10.622954 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.622941    2581 manager.go:349] Starting recovery of all containers
Jul 28 17:09:10.624080 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:10.624006    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 17:09:10.624080 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:10.623973    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-231.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 17:09:10.636395 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.636378    2581 manager.go:354] Recovery completed
Jul 28 17:09:10.640174 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.640056    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-djqw5"
Jul 28 17:09:10.644017 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.643989    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:09:10.647934 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.647915    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-djqw5"
Jul 28 17:09:10.648779 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.648765    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:09:10.648851 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.648797    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:09:10.648851 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.648811    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:09:10.649320 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.649305    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 17:09:10.649396 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.649346    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 17:09:10.649396 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.649358    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:09:10.649396 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.649365    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:09:10.650247 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.650229    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:09:10.650315 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.650256    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:09:10.650315 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.650265    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:09:10.652648 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.652633    2581 policy_none.go:49] "None policy: Start"
Jul 28 17:09:10.652715 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.652653    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 17:09:10.652715 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.652666    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 17:09:10.654582 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.654571    2581 policy_none.go:47] "Start"
Jul 28 17:09:10.702190 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.691161    2581 manager.go:337] "Starting Device Plugin manager"
Jul 28 17:09:10.702190 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:10.691283    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 17:09:10.702190 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.691293    2581 server.go:87] "Starting device plugin registration server"
Jul 28 17:09:10.702190 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.691535    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 17:09:10.702190 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.691549    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 17:09:10.702190 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.691691    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 17:09:10.702190 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.691768    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 17:09:10.702190 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.691776    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 17:09:10.702190 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:10.692164    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 17:09:10.702190 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:10.692208    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-231.ec2.internal\" not found"
Jul 28 17:09:10.767096 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.767071    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 17:09:10.768152 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.768130    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 17:09:10.768246 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.768171    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 17:09:10.768246 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.768194    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 17:09:10.768246 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:10.768231    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 17:09:10.770264 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.770246    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:09:10.791893 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.791850    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:09:10.792624 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.792590    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:09:10.792700 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.792636    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:09:10.792700 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.792650    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:09:10.792700 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.792673    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-231.ec2.internal"
Jul 28 17:09:10.803095 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.803080    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-231.ec2.internal"
Jul 28 17:09:10.803136 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:10.803100    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-231.ec2.internal\": node \"ip-10-0-138-231.ec2.internal\" not found"
Jul 28 17:09:10.823972 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:10.823947    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-231.ec2.internal\" not found"
Jul 28 17:09:10.868319 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.868291    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-231.ec2.internal"]
Jul 28 17:09:10.868400 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.868356    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:09:10.869748 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.869736    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:09:10.869811 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.869760    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:09:10.869811 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.869769    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:09:10.872131 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.872119    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:09:10.872254 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.872243    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal"
Jul 28 17:09:10.872289 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.872271    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:09:10.872823 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.872807    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:09:10.872903 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.872832    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:09:10.872903 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.872841    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:09:10.872903 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.872894    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:09:10.873009 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.872917    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:09:10.873009 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.872928    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:09:10.875080 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.875064    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-231.ec2.internal"
Jul 28 17:09:10.875150 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.875088    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:09:10.875764 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.875748    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:09:10.875837 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.875774    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:09:10.875837 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:10.875785    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:09:10.898235 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:10.898214    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-231.ec2.internal\" not found" node="ip-10-0-138-231.ec2.internal"
Jul 28 17:09:10.902502 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:10.902488    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-231.ec2.internal\" not found" node="ip-10-0-138-231.ec2.internal"
Jul 28 17:09:10.924373 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:10.924357    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-231.ec2.internal\" not found"
Jul 28 17:09:11.024275 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:11.024248    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d6006a44a5831824d4f450fa2922581d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal\" (UID: \"d6006a44a5831824d4f450fa2922581d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal"
Jul 28 17:09:11.024275 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:11.024276    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6006a44a5831824d4f450fa2922581d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal\" (UID: \"d6006a44a5831824d4f450fa2922581d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal"
Jul 28 17:09:11.024412 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:11.024291    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/83f9b4a12578fa1633b19325b5319dc4-config\") pod \"kube-apiserver-proxy-ip-10-0-138-231.ec2.internal\" (UID: \"83f9b4a12578fa1633b19325b5319dc4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-231.ec2.internal"
Jul 28 17:09:11.025293 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:11.025278    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-231.ec2.internal\" not found"
Jul 28 17:09:11.125001 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:11.124947    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/83f9b4a12578fa1633b19325b5319dc4-config\") pod \"kube-apiserver-proxy-ip-10-0-138-231.ec2.internal\" (UID: \"83f9b4a12578fa1633b19325b5319dc4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-231.ec2.internal"
Jul 28 17:09:11.125001 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:11.124975    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d6006a44a5831824d4f450fa2922581d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal\" (UID: \"d6006a44a5831824d4f450fa2922581d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal"
Jul 28 17:09:11.125001 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:11.124993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6006a44a5831824d4f450fa2922581d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal\" (UID: \"d6006a44a5831824d4f450fa2922581d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal"
Jul 28 17:09:11.125136 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:11.125035    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6006a44a5831824d4f450fa2922581d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal\" (UID: \"d6006a44a5831824d4f450fa2922581d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal"
Jul 28 17:09:11.125136 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:11.125057    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/83f9b4a12578fa1633b19325b5319dc4-config\") pod \"kube-apiserver-proxy-ip-10-0-138-231.ec2.internal\" (UID: \"83f9b4a12578fa1633b19325b5319dc4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-231.ec2.internal"
Jul 28 17:09:11.125136 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:11.125064    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d6006a44a5831824d4f450fa2922581d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal\" (UID: \"d6006a44a5831824d4f450fa2922581d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal"
Jul 28 17:09:11.126011 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:11.125996    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-231.ec2.internal\" not found"
Jul 28 17:09:11.201204 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:11.201177    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal"
Jul 28 17:09:11.205548 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:11.205521    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-231.ec2.internal"
Jul 28 17:09:11.227586 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:11.227565    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-231.ec2.internal\" not found"
Jul 28 17:09:11.328150 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:11.328118    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-231.ec2.internal\" not found"
Jul 28 17:09:11.428651 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:11.428592    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-231.ec2.internal\" not found"
Jul 28 17:09:11.494005 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:11.493980    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:09:11.528888 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:11.528864    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-231.ec2.internal\" not found"
Jul 28 17:09:11.553035 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:11.553015    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 17:09:11.553143 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:11.553126    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:09:11.553180 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:11.553161    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:09:11.629797 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:11.629775    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-231.ec2.internal\" not found"
Jul 28 17:09:11.654667 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:11.654621    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 17:04:12 +0000 UTC" deadline="2028-01-04 02:42:30.018228738 +0000 UTC"
Jul 28 17:09:11.654667 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:11.654659    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12585h33m18.363574415s"
Jul 28 17:09:11.730267 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:11.730218    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-231.ec2.internal\" not found"
Jul 28 17:09:13.268084 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:13.268052    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f9b4a12578fa1633b19325b5319dc4.slice/crio-0daee3eb83265628c0810dff1fe40d89aaabab827561e3dbcb5ac45afed266e0 WatchSource:0}: Error finding container 0daee3eb83265628c0810dff1fe40d89aaabab827561e3dbcb5ac45afed266e0: Status 404 returned error can't find the container with id 0daee3eb83265628c0810dff1fe40d89aaabab827561e3dbcb5ac45afed266e0
Jul 28 17:09:13.268342 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:13.268307    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6006a44a5831824d4f450fa2922581d.slice/crio-3ada9c7876f1809c40a1eae9d0ea166ac2c69c735695bf1f7c93051fe9d77927 WatchSource:0}: Error finding container 3ada9c7876f1809c40a1eae9d0ea166ac2c69c735695bf1f7c93051fe9d77927: Status 404 returned error can't find the container with id 3ada9c7876f1809c40a1eae9d0ea166ac2c69c735695bf1f7c93051fe9d77927
Jul 28 17:09:13.271871 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:13.271859    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:09:13.287411 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:13.287374    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-231.ec2.internal" event={"ID":"83f9b4a12578fa1633b19325b5319dc4","Type":"ContainerStarted","Data":"0daee3eb83265628c0810dff1fe40d89aaabab827561e3dbcb5ac45afed266e0"}
Jul 28 17:09:13.288381 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:13.288362    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal" event={"ID":"d6006a44a5831824d4f450fa2922581d","Type":"ContainerStarted","Data":"3ada9c7876f1809c40a1eae9d0ea166ac2c69c735695bf1f7c93051fe9d77927"}
Jul 28 17:09:13.346907 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:13.346858    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-231.ec2.internal\" not found"
Jul 28 17:09:13.447422 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:13.447400    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-231.ec2.internal\" not found"
Jul 28 17:09:13.547920 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:13.547854    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-231.ec2.internal\" not found"
Jul 28 17:09:13.577136 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:13.577112    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:09:13.582558 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:13.581633    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:09:13.637876 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:13.637848    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal"
Jul 28 17:09:13.653018 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:13.653000    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:09:13.653769 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:13.653755    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-231.ec2.internal"
Jul 28 17:09:13.666697 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:13.666672    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:09:14.029410 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.029338    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:09:14.110592 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.110567    2581 apiserver.go:52] "Watching apiserver"
Jul 28 17:09:14.119117 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.119092    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 17:09:14.119514 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.119486    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-2nt29","openshift-network-operator/iptables-alerter-pp6xv","kube-system/global-pull-secret-syncer-jwmpm","openshift-dns/node-resolver-dbnhf","openshift-image-registry/node-ca-ffvhq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal","openshift-multus/multus-additional-cni-plugins-njk8v","openshift-multus/multus-r5x5j","openshift-multus/network-metrics-daemon-zkf8x","openshift-ovn-kubernetes/ovnkube-node-9l7dh","kube-system/konnectivity-agent-kc9pw","kube-system/kube-apiserver-proxy-ip-10-0-138-231.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg","openshift-cluster-node-tuning-operator/tuned-jhs8c"]
Jul 28 17:09:14.123016 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.122131    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-njk8v"
Jul 28 17:09:14.124451 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.124425    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg"
Jul 28 17:09:14.127288 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.126552    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:14.127288 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:14.126611    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwmpm" podUID="189dbf08-647d-42a8-975f-39838ad4edf9"
Jul 28 17:09:14.129117 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.128371    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 17:09:14.129117 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.128738    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dbnhf"
Jul 28 17:09:14.129689 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.129363    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 17:09:14.129689 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.129442    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 17:09:14.129689 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.129493    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 17:09:14.129689 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.129442    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 17:09:14.129982 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.129964    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 17:09:14.130199 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.130184    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 17:09:14.131296 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.131229    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ffvhq"
Jul 28 17:09:14.131887 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.131766    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 17:09:14.131887 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.131824    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p7nl2\""
Jul 28 17:09:14.132047 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.132031    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f7kdf\""
Jul 28 17:09:14.132781 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.132608    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 17:09:14.133756 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.133329    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.133756 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.133330    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 17:09:14.135897 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.135635    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:14.135897 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:14.135708    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkf8x" podUID="9a7b47e4-993e-4527-a00f-4ebbbc050348"
Jul 28 17:09:14.138195 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.138166    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 17:09:14.138307 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.138297    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 17:09:14.138664 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.138533    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 17:09:14.138988 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.138959    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z5p7b\""
Jul 28 17:09:14.139079 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.139053    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 17:09:14.139289 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.139256    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:09:14.139372 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:14.139333    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nt29" podUID="33068506-3820-4d64-bc6b-33ea180dca70"
Jul 28 17:09:14.140710 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.140190    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pp6xv"
Jul 28 17:09:14.142115 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.141876    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:09:14.142115 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.141891    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 17:09:14.142115 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.141901    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-l4cwr\""
Jul 28 17:09:14.142115 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.142048    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2mmw2\""
Jul 28 17:09:14.143497 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.142614    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.147410 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.146251    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kc9pw"
Jul 28 17:09:14.148907 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.148888    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.149237 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.149212    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c643884f-420e-452c-98b0-09aede5dd675-sys-fs\") pod \"aws-ebs-csi-driver-node-t82zg\" (UID: \"c643884f-420e-452c-98b0-09aede5dd675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg"
Jul 28 17:09:14.149312 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.149253    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fj75\" (UniqueName: \"kubernetes.io/projected/8188b00b-a96c-4bd3-a9b9-35fc129184d5-kube-api-access-8fj75\") pod \"iptables-alerter-pp6xv\" (UID: \"8188b00b-a96c-4bd3-a9b9-35fc129184d5\") " pod="openshift-network-operator/iptables-alerter-pp6xv"
Jul 28 17:09:14.149312 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.149281    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e459f08-3f93-48bd-bb45-b950f416c87f-system-cni-dir\") pod \"multus-additional-cni-plugins-njk8v\" (UID: \"1e459f08-3f93-48bd-bb45-b950f416c87f\") " pod="openshift-multus/multus-additional-cni-plugins-njk8v"
Jul 28 17:09:14.149312 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.149303    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c643884f-420e-452c-98b0-09aede5dd675-device-dir\") pod \"aws-ebs-csi-driver-node-t82zg\" (UID: \"c643884f-420e-452c-98b0-09aede5dd675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg"
Jul 28 17:09:14.149468 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.149338    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a7b47e4-993e-4527-a00f-4ebbbc050348-metrics-certs\") pod \"network-metrics-daemon-zkf8x\" (UID: \"9a7b47e4-993e-4527-a00f-4ebbbc050348\") " pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:14.149468 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.149361    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e459f08-3f93-48bd-bb45-b950f416c87f-cni-binary-copy\") pod \"multus-additional-cni-plugins-njk8v\" (UID: \"1e459f08-3f93-48bd-bb45-b950f416c87f\") " pod="openshift-multus/multus-additional-cni-plugins-njk8v"
Jul 28 17:09:14.149468 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.149377    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ed21aa75-f994-4548-93f7-99c172998ea1-etc-tuned\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.149468 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.149397    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsw86\" (UniqueName: \"kubernetes.io/projected/ed21aa75-f994-4548-93f7-99c172998ea1-kube-api-access-jsw86\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.149468 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.149435    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nrfd\" (UniqueName: \"kubernetes.io/projected/9a7b47e4-993e-4527-a00f-4ebbbc050348-kube-api-access-5nrfd\") pod \"network-metrics-daemon-zkf8x\" (UID: \"9a7b47e4-993e-4527-a00f-4ebbbc050348\") " pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:14.149702 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.149468    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1e459f08-3f93-48bd-bb45-b950f416c87f-os-release\") pod \"multus-additional-cni-plugins-njk8v\" (UID: \"1e459f08-3f93-48bd-bb45-b950f416c87f\") " pod="openshift-multus/multus-additional-cni-plugins-njk8v"
Jul 28 17:09:14.149702 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.149490    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e459f08-3f93-48bd-bb45-b950f416c87f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-njk8v\" (UID: \"1e459f08-3f93-48bd-bb45-b950f416c87f\") " pod="openshift-multus/multus-additional-cni-plugins-njk8v"
Jul 28 17:09:14.149702 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.149540    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1e459f08-3f93-48bd-bb45-b950f416c87f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-njk8v\" (UID: \"1e459f08-3f93-48bd-bb45-b950f416c87f\") " pod="openshift-multus/multus-additional-cni-plugins-njk8v"
Jul 28 17:09:14.149702 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.149568    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/189dbf08-647d-42a8-975f-39838ad4edf9-kubelet-config\") pod \"global-pull-secret-syncer-jwmpm\" (UID: \"189dbf08-647d-42a8-975f-39838ad4edf9\") " pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:14.149702 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.149584    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2bvc\" (UniqueName: \"kubernetes.io/projected/601b853c-8800-428b-89f6-1cb7b378600c-kube-api-access-r2bvc\") pod \"node-resolver-dbnhf\" (UID: \"601b853c-8800-428b-89f6-1cb7b378600c\") " pod="openshift-dns/node-resolver-dbnhf"
Jul 28 17:09:14.149702 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.149604    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-lib-modules\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.149702 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.149620    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c643884f-420e-452c-98b0-09aede5dd675-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t82zg\" (UID: \"c643884f-420e-452c-98b0-09aede5dd675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg"
Jul 28 17:09:14.149702 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.149638    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c643884f-420e-452c-98b0-09aede5dd675-registration-dir\") pod \"aws-ebs-csi-driver-node-t82zg\" (UID: \"c643884f-420e-452c-98b0-09aede5dd675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg"
Jul 28 17:09:14.149702 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.149660    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svdn6\" (UniqueName: \"kubernetes.io/projected/6d4cfcb5-839a-431f-946c-6b5ac4e6c61e-kube-api-access-svdn6\") pod \"node-ca-ffvhq\" (UID: \"6d4cfcb5-839a-431f-946c-6b5ac4e6c61e\") " pod="openshift-image-registry/node-ca-ffvhq"
Jul 28 17:09:14.150133 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.149734    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8188b00b-a96c-4bd3-a9b9-35fc129184d5-host-slash\") pod \"iptables-alerter-pp6xv\" (UID: \"8188b00b-a96c-4bd3-a9b9-35fc129184d5\") " pod="openshift-network-operator/iptables-alerter-pp6xv"
Jul 28 17:09:14.150133 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.149787    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e459f08-3f93-48bd-bb45-b950f416c87f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-njk8v\" (UID: \"1e459f08-3f93-48bd-bb45-b950f416c87f\") " pod="openshift-multus/multus-additional-cni-plugins-njk8v"
Jul 28 17:09:14.150133 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.149823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/189dbf08-647d-42a8-975f-39838ad4edf9-original-pull-secret\") pod \"global-pull-secret-syncer-jwmpm\" (UID: \"189dbf08-647d-42a8-975f-39838ad4edf9\") " pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:14.150133 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.149847    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/601b853c-8800-428b-89f6-1cb7b378600c-hosts-file\") pod \"node-resolver-dbnhf\" (UID: \"601b853c-8800-428b-89f6-1cb7b378600c\") " pod="openshift-dns/node-resolver-dbnhf"
Jul 28 17:09:14.150133 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.149871    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-etc-sysctl-conf\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.150133 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.149951    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6d4cfcb5-839a-431f-946c-6b5ac4e6c61e-serviceca\") pod \"node-ca-ffvhq\" (UID: \"6d4cfcb5-839a-431f-946c-6b5ac4e6c61e\") " pod="openshift-image-registry/node-ca-ffvhq"
Jul 28 17:09:14.150133 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.149986    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/601b853c-8800-428b-89f6-1cb7b378600c-tmp-dir\") pod \"node-resolver-dbnhf\" (UID: \"601b853c-8800-428b-89f6-1cb7b378600c\") " pod="openshift-dns/node-resolver-dbnhf"
Jul 28 17:09:14.150133 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.150009    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-etc-modprobe-d\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.150133 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.150031    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-host\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.150133 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.150070    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c643884f-420e-452c-98b0-09aede5dd675-socket-dir\") pod \"aws-ebs-csi-driver-node-t82zg\" (UID: \"c643884f-420e-452c-98b0-09aede5dd675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg"
Jul 28 17:09:14.150133 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.150100    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8188b00b-a96c-4bd3-a9b9-35fc129184d5-iptables-alerter-script\") pod \"iptables-alerter-pp6xv\" (UID: \"8188b00b-a96c-4bd3-a9b9-35fc129184d5\") " pod="openshift-network-operator/iptables-alerter-pp6xv"
Jul 28 17:09:14.150133 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.150123    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b89l\" (UniqueName: \"kubernetes.io/projected/1e459f08-3f93-48bd-bb45-b950f416c87f-kube-api-access-7b89l\") pod \"multus-additional-cni-plugins-njk8v\" (UID: \"1e459f08-3f93-48bd-bb45-b950f416c87f\") " pod="openshift-multus/multus-additional-cni-plugins-njk8v"
Jul 28 17:09:14.150677 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.150148    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-etc-sysconfig\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.150677 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.150171    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-etc-kubernetes\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.150677 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.150194    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-sys\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.150677 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.150223    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-var-lib-kubelet\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.150677 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.150250    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed21aa75-f994-4548-93f7-99c172998ea1-tmp\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.150677 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.150275    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfjsz\" (UniqueName: \"kubernetes.io/projected/c643884f-420e-452c-98b0-09aede5dd675-kube-api-access-lfjsz\") pod \"aws-ebs-csi-driver-node-t82zg\" (UID: \"c643884f-420e-452c-98b0-09aede5dd675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg"
Jul 28 17:09:14.150677 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.150296    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-etc-sysctl-d\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.150677 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.150316    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-etc-systemd\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.150677 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.150349    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-run\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.150677 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.150371    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgzn7\" (UniqueName: \"kubernetes.io/projected/33068506-3820-4d64-bc6b-33ea180dca70-kube-api-access-jgzn7\") pod \"network-check-target-2nt29\" (UID: \"33068506-3820-4d64-bc6b-33ea180dca70\") " pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:09:14.150677 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.150392    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c643884f-420e-452c-98b0-09aede5dd675-etc-selinux\") pod \"aws-ebs-csi-driver-node-t82zg\" (UID: \"c643884f-420e-452c-98b0-09aede5dd675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg"
Jul 28 17:09:14.150677 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.150413    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d4cfcb5-839a-431f-946c-6b5ac4e6c61e-host\") pod \"node-ca-ffvhq\" (UID: \"6d4cfcb5-839a-431f-946c-6b5ac4e6c61e\") " pod="openshift-image-registry/node-ca-ffvhq"
Jul 28 17:09:14.150677 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.150434    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e459f08-3f93-48bd-bb45-b950f416c87f-cnibin\") pod \"multus-additional-cni-plugins-njk8v\" (UID: \"1e459f08-3f93-48bd-bb45-b950f416c87f\") " pod="openshift-multus/multus-additional-cni-plugins-njk8v"
Jul 28 17:09:14.150677 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.150454    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/189dbf08-647d-42a8-975f-39838ad4edf9-dbus\") pod \"global-pull-secret-syncer-jwmpm\" (UID: \"189dbf08-647d-42a8-975f-39838ad4edf9\") " pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:14.164345 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.164176    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 17:09:14.164986 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.164964    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 17:09:14.164986 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.164977    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 17:09:14.165179 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.165161    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 17:09:14.165315 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.165226    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 17:09:14.165377 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.165325    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mlhjz\""
Jul 28 17:09:14.165377 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.165337    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 17:09:14.165476 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.165375    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 17:09:14.165476 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.165433    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 17:09:14.165607 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.165590    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 17:09:14.165673 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.165636    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-24vm7\""
Jul 28 17:09:14.165673 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.165644    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 17:09:14.165772 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.165647    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 17:09:14.165772 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.165741    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-49g7g\""
Jul 28 17:09:14.166440 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.166420    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2v4dj\""
Jul 28 17:09:14.171050 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.171034    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:09:14.179500 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.179478    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:09:14.239633 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.239599    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 17:09:14.251336 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.251308    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b0a6ecd5-1b96-4003-ba79-fe147a67c757-ovnkube-script-lib\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.251470 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.251347    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5a4cb725-44da-4042-8bc6-8f09251a6325-multus-daemon-config\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.251470 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.251372    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-etc-openvswitch\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.251470 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.251434    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c643884f-420e-452c-98b0-09aede5dd675-device-dir\") pod \"aws-ebs-csi-driver-node-t82zg\" (UID: \"c643884f-420e-452c-98b0-09aede5dd675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg"
Jul 28 17:09:14.251591 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.251473    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e459f08-3f93-48bd-bb45-b950f416c87f-cni-binary-copy\") pod \"multus-additional-cni-plugins-njk8v\" (UID: \"1e459f08-3f93-48bd-bb45-b950f416c87f\") " pod="openshift-multus/multus-additional-cni-plugins-njk8v"
Jul 28 17:09:14.251591 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.251506    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-host-kubelet\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.251591 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.251530    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-host-slash\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.251591 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.251554    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-var-lib-openvswitch\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.251591 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.251578    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-system-cni-dir\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.251841 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.251603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nrfd\" (UniqueName: \"kubernetes.io/projected/9a7b47e4-993e-4527-a00f-4ebbbc050348-kube-api-access-5nrfd\") pod \"network-metrics-daemon-zkf8x\" (UID: \"9a7b47e4-993e-4527-a00f-4ebbbc050348\") " pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:14.251841 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.251627    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1e459f08-3f93-48bd-bb45-b950f416c87f-os-release\") pod \"multus-additional-cni-plugins-njk8v\" (UID: \"1e459f08-3f93-48bd-bb45-b950f416c87f\") " pod="openshift-multus/multus-additional-cni-plugins-njk8v"
Jul 28 17:09:14.251841 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.251656    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2bvc\" (UniqueName: \"kubernetes.io/projected/601b853c-8800-428b-89f6-1cb7b378600c-kube-api-access-r2bvc\") pod \"node-resolver-dbnhf\" (UID: \"601b853c-8800-428b-89f6-1cb7b378600c\") " pod="openshift-dns/node-resolver-dbnhf"
Jul 28 17:09:14.251841 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.251694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-lib-modules\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.251841 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.251726    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-host-run-netns\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.251841 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.251749    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b0a6ecd5-1b96-4003-ba79-fe147a67c757-env-overrides\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.251841 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.251776    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c643884f-420e-452c-98b0-09aede5dd675-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t82zg\" (UID: \"c643884f-420e-452c-98b0-09aede5dd675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg"
Jul 28 17:09:14.251841 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.251798    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c643884f-420e-452c-98b0-09aede5dd675-device-dir\") pod \"aws-ebs-csi-driver-node-t82zg\" (UID: \"c643884f-420e-452c-98b0-09aede5dd675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg"
Jul 28 17:09:14.252302 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.251853    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8188b00b-a96c-4bd3-a9b9-35fc129184d5-host-slash\") pod \"iptables-alerter-pp6xv\" (UID: \"8188b00b-a96c-4bd3-a9b9-35fc129184d5\") " pod="openshift-network-operator/iptables-alerter-pp6xv"
Jul 28 17:09:14.252302 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.251801    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8188b00b-a96c-4bd3-a9b9-35fc129184d5-host-slash\") pod \"iptables-alerter-pp6xv\" (UID: \"8188b00b-a96c-4bd3-a9b9-35fc129184d5\") " pod="openshift-network-operator/iptables-alerter-pp6xv"
Jul 28 17:09:14.252302 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.251901    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e459f08-3f93-48bd-bb45-b950f416c87f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-njk8v\" (UID: \"1e459f08-3f93-48bd-bb45-b950f416c87f\") " pod="openshift-multus/multus-additional-cni-plugins-njk8v"
Jul 28 17:09:14.252302 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.251958    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/601b853c-8800-428b-89f6-1cb7b378600c-hosts-file\") pod \"node-resolver-dbnhf\" (UID: \"601b853c-8800-428b-89f6-1cb7b378600c\") " pod="openshift-dns/node-resolver-dbnhf"
Jul 28 17:09:14.252302 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.251989    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-run-systemd\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.252302 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252002    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-lib-modules\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.252302 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252028    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e459f08-3f93-48bd-bb45-b950f416c87f-cni-binary-copy\") pod \"multus-additional-cni-plugins-njk8v\" (UID: \"1e459f08-3f93-48bd-bb45-b950f416c87f\") " pod="openshift-multus/multus-additional-cni-plugins-njk8v"
Jul 28 17:09:14.252302 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252041    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-host-run-ovn-kubernetes\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.252302 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252119    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-host-cni-netd\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.252302 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252146    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c643884f-420e-452c-98b0-09aede5dd675-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t82zg\" (UID: \"c643884f-420e-452c-98b0-09aede5dd675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg"
Jul 28 17:09:14.252302 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252155    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1e459f08-3f93-48bd-bb45-b950f416c87f-os-release\") pod \"multus-additional-cni-plugins-njk8v\" (UID: \"1e459f08-3f93-48bd-bb45-b950f416c87f\") " pod="openshift-multus/multus-additional-cni-plugins-njk8v"
Jul 28 17:09:14.252302 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252231    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnhp8\" (UniqueName: \"kubernetes.io/projected/b0a6ecd5-1b96-4003-ba79-fe147a67c757-kube-api-access-mnhp8\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.252302 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252261    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e459f08-3f93-48bd-bb45-b950f416c87f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-njk8v\" (UID: \"1e459f08-3f93-48bd-bb45-b950f416c87f\") " pod="openshift-multus/multus-additional-cni-plugins-njk8v"
Jul 28 17:09:14.252302 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252276    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6d4cfcb5-839a-431f-946c-6b5ac4e6c61e-serviceca\") pod \"node-ca-ffvhq\" (UID: \"6d4cfcb5-839a-431f-946c-6b5ac4e6c61e\") " pod="openshift-image-registry/node-ca-ffvhq"
Jul 28 17:09:14.252980 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252311    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/601b853c-8800-428b-89f6-1cb7b378600c-hosts-file\") pod \"node-resolver-dbnhf\" (UID: \"601b853c-8800-428b-89f6-1cb7b378600c\") " pod="openshift-dns/node-resolver-dbnhf"
Jul 28 17:09:14.252980 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/601b853c-8800-428b-89f6-1cb7b378600c-tmp-dir\") pod \"node-resolver-dbnhf\" (UID: \"601b853c-8800-428b-89f6-1cb7b378600c\") " pod="openshift-dns/node-resolver-dbnhf"
Jul 28 17:09:14.252980 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252373    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-host\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.252980 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252416    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-host\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.252980 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252444    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-host-var-lib-cni-multus\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.252980 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252484    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8188b00b-a96c-4bd3-a9b9-35fc129184d5-iptables-alerter-script\") pod \"iptables-alerter-pp6xv\" (UID: \"8188b00b-a96c-4bd3-a9b9-35fc129184d5\") " pod="openshift-network-operator/iptables-alerter-pp6xv"
Jul 28 17:09:14.252980 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-etc-sysconfig\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.252980 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252617    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed21aa75-f994-4548-93f7-99c172998ea1-tmp\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.252980 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252652    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-systemd-units\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.252980 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252655    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-etc-sysconfig\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.252980 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252664    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/601b853c-8800-428b-89f6-1cb7b378600c-tmp-dir\") pod \"node-resolver-dbnhf\" (UID: \"601b853c-8800-428b-89f6-1cb7b378600c\") " pod="openshift-dns/node-resolver-dbnhf"
Jul 28 17:09:14.252980 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252680    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5a4cb725-44da-4042-8bc6-8f09251a6325-cni-binary-copy\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.252980 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252705    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6d4cfcb5-839a-431f-946c-6b5ac4e6c61e-serviceca\") pod \"node-ca-ffvhq\" (UID: \"6d4cfcb5-839a-431f-946c-6b5ac4e6c61e\") " pod="openshift-image-registry/node-ca-ffvhq"
Jul 28 17:09:14.252980 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252720    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8md76\" (UniqueName: \"kubernetes.io/projected/5a4cb725-44da-4042-8bc6-8f09251a6325-kube-api-access-8md76\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.252980 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252756    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfjsz\" (UniqueName: \"kubernetes.io/projected/c643884f-420e-452c-98b0-09aede5dd675-kube-api-access-lfjsz\") pod \"aws-ebs-csi-driver-node-t82zg\" (UID: \"c643884f-420e-452c-98b0-09aede5dd675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg"
Jul 28 17:09:14.252980 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252826    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-etc-systemd\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.252980 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252844    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-run\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.252980 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252868    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgzn7\" (UniqueName: \"kubernetes.io/projected/33068506-3820-4d64-bc6b-33ea180dca70-kube-api-access-jgzn7\") pod \"network-check-target-2nt29\" (UID: \"33068506-3820-4d64-bc6b-33ea180dca70\") " pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:09:14.253777 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252893    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-node-log\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.253777 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252899    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-etc-systemd\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.253777 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252917    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-host-cni-bin\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.253777 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.252895    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 17:09:14.253777 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.253053    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8188b00b-a96c-4bd3-a9b9-35fc129184d5-iptables-alerter-script\") pod \"iptables-alerter-pp6xv\" (UID: \"8188b00b-a96c-4bd3-a9b9-35fc129184d5\") " pod="openshift-network-operator/iptables-alerter-pp6xv"
Jul 28 17:09:14.253777 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.253061    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-multus-cni-dir\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.253777 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.253070    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-run\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.253777 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.253100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c643884f-420e-452c-98b0-09aede5dd675-etc-selinux\") pod \"aws-ebs-csi-driver-node-t82zg\" (UID: \"c643884f-420e-452c-98b0-09aede5dd675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg"
Jul 28 17:09:14.253777 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.253122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e459f08-3f93-48bd-bb45-b950f416c87f-cnibin\") pod \"multus-additional-cni-plugins-njk8v\" (UID: \"1e459f08-3f93-48bd-bb45-b950f416c87f\") " pod="openshift-multus/multus-additional-cni-plugins-njk8v"
Jul 28 17:09:14.253777 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.253137    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-cnibin\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.253777 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.253173    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-multus-socket-dir-parent\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.253777 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.253204    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c643884f-420e-452c-98b0-09aede5dd675-etc-selinux\") pod \"aws-ebs-csi-driver-node-t82zg\" (UID: \"c643884f-420e-452c-98b0-09aede5dd675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg"
Jul 28 17:09:14.253777 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.253203    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-host-run-k8s-cni-cncf-io\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.253777 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.253212    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e459f08-3f93-48bd-bb45-b950f416c87f-cnibin\") pod \"multus-additional-cni-plugins-njk8v\" (UID: \"1e459f08-3f93-48bd-bb45-b950f416c87f\") " pod="openshift-multus/multus-additional-cni-plugins-njk8v"
Jul 28 17:09:14.253777 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.253231    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-host-var-lib-kubelet\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.253777 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.253270    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-etc-kubernetes\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.253777 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.253299    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3b4343ba-84d3-4d95-8a25-e97b6cf88830-konnectivity-ca\") pod \"konnectivity-agent-kc9pw\" (UID: \"3b4343ba-84d3-4d95-8a25-e97b6cf88830\") " pod="kube-system/konnectivity-agent-kc9pw"
Jul 28 17:09:14.254518 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.253326    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c643884f-420e-452c-98b0-09aede5dd675-sys-fs\") pod \"aws-ebs-csi-driver-node-t82zg\" (UID: \"c643884f-420e-452c-98b0-09aede5dd675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg"
Jul 28 17:09:14.254518 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.253353    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fj75\" (UniqueName: \"kubernetes.io/projected/8188b00b-a96c-4bd3-a9b9-35fc129184d5-kube-api-access-8fj75\") pod \"iptables-alerter-pp6xv\" (UID: \"8188b00b-a96c-4bd3-a9b9-35fc129184d5\") " pod="openshift-network-operator/iptables-alerter-pp6xv"
Jul 28 17:09:14.254518 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.253382    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e459f08-3f93-48bd-bb45-b950f416c87f-system-cni-dir\") pod \"multus-additional-cni-plugins-njk8v\" (UID: \"1e459f08-3f93-48bd-bb45-b950f416c87f\") " pod="openshift-multus/multus-additional-cni-plugins-njk8v"
Jul 28 17:09:14.254518 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.253408    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-run-ovn\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.254518 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.253386    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c643884f-420e-452c-98b0-09aede5dd675-sys-fs\") pod \"aws-ebs-csi-driver-node-t82zg\" (UID: \"c643884f-420e-452c-98b0-09aede5dd675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg"
Jul 28 17:09:14.254518 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.253434    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0a6ecd5-1b96-4003-ba79-fe147a67c757-ovn-node-metrics-cert\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.254518 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.253456    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-multus-conf-dir\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.254518 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.253472    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e459f08-3f93-48bd-bb45-b950f416c87f-system-cni-dir\") pod \"multus-additional-cni-plugins-njk8v\" (UID: \"1e459f08-3f93-48bd-bb45-b950f416c87f\") " pod="openshift-multus/multus-additional-cni-plugins-njk8v"
Jul 28 17:09:14.254518 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.253499    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a7b47e4-993e-4527-a00f-4ebbbc050348-metrics-certs\") pod \"network-metrics-daemon-zkf8x\" (UID: \"9a7b47e4-993e-4527-a00f-4ebbbc050348\") " pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:14.254518 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.253528    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ed21aa75-f994-4548-93f7-99c172998ea1-etc-tuned\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.254518 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:14.253576    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:14.254518 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:14.253650    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a7b47e4-993e-4527-a00f-4ebbbc050348-metrics-certs podName:9a7b47e4-993e-4527-a00f-4ebbbc050348 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:14.753616038 +0000 UTC m=+3.028232456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a7b47e4-993e-4527-a00f-4ebbbc050348-metrics-certs") pod "network-metrics-daemon-zkf8x" (UID: "9a7b47e4-993e-4527-a00f-4ebbbc050348") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:14.254518 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsw86\" (UniqueName: \"kubernetes.io/projected/ed21aa75-f994-4548-93f7-99c172998ea1-kube-api-access-jsw86\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.254518 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254054    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-log-socket\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.254518 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254080    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-os-release\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.254518 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254103    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-run-openvswitch\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.255262 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254126    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e459f08-3f93-48bd-bb45-b950f416c87f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-njk8v\" (UID: \"1e459f08-3f93-48bd-bb45-b950f416c87f\") " pod="openshift-multus/multus-additional-cni-plugins-njk8v"
Jul 28 17:09:14.255262 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254143    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1e459f08-3f93-48bd-bb45-b950f416c87f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-njk8v\" (UID: \"1e459f08-3f93-48bd-bb45-b950f416c87f\") " pod="openshift-multus/multus-additional-cni-plugins-njk8v"
Jul 28 17:09:14.255262 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254160    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/189dbf08-647d-42a8-975f-39838ad4edf9-kubelet-config\") pod \"global-pull-secret-syncer-jwmpm\" (UID: \"189dbf08-647d-42a8-975f-39838ad4edf9\") " pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:14.255262 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254207    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/189dbf08-647d-42a8-975f-39838ad4edf9-kubelet-config\") pod \"global-pull-secret-syncer-jwmpm\" (UID: \"189dbf08-647d-42a8-975f-39838ad4edf9\") " pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:14.255262 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254220    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c643884f-420e-452c-98b0-09aede5dd675-registration-dir\") pod \"aws-ebs-csi-driver-node-t82zg\" (UID: \"c643884f-420e-452c-98b0-09aede5dd675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg"
Jul 28 17:09:14.255262 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254249    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svdn6\" (UniqueName: \"kubernetes.io/projected/6d4cfcb5-839a-431f-946c-6b5ac4e6c61e-kube-api-access-svdn6\") pod \"node-ca-ffvhq\" (UID: \"6d4cfcb5-839a-431f-946c-6b5ac4e6c61e\") " pod="openshift-image-registry/node-ca-ffvhq"
Jul 28 17:09:14.255262 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254280    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/189dbf08-647d-42a8-975f-39838ad4edf9-original-pull-secret\") pod \"global-pull-secret-syncer-jwmpm\" (UID: \"189dbf08-647d-42a8-975f-39838ad4edf9\") " pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:14.255262 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254304    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-etc-sysctl-conf\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.255262 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254332    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-host-run-netns\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.255262 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:14.254403    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:14.255262 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:14.254458    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/189dbf08-647d-42a8-975f-39838ad4edf9-original-pull-secret podName:189dbf08-647d-42a8-975f-39838ad4edf9 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:14.75444201 +0000 UTC m=+3.029058425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/189dbf08-647d-42a8-975f-39838ad4edf9-original-pull-secret") pod "global-pull-secret-syncer-jwmpm" (UID: "189dbf08-647d-42a8-975f-39838ad4edf9") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:14.255262 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254531    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c643884f-420e-452c-98b0-09aede5dd675-registration-dir\") pod \"aws-ebs-csi-driver-node-t82zg\" (UID: \"c643884f-420e-452c-98b0-09aede5dd675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg"
Jul 28 17:09:14.255262 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254563    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-hostroot\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.255262 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254574    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-etc-sysctl-conf\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.255262 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254592    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-host-run-multus-certs\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.255262 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254617    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-etc-modprobe-d\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.255957 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254632    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-host-var-lib-cni-bin\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.255957 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254643    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e459f08-3f93-48bd-bb45-b950f416c87f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-njk8v\" (UID: \"1e459f08-3f93-48bd-bb45-b950f416c87f\") " pod="openshift-multus/multus-additional-cni-plugins-njk8v"
Jul 28 17:09:14.255957 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254649    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c643884f-420e-452c-98b0-09aede5dd675-socket-dir\") pod \"aws-ebs-csi-driver-node-t82zg\" (UID: \"c643884f-420e-452c-98b0-09aede5dd675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg"
Jul 28 17:09:14.255957 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254714    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b89l\" (UniqueName: \"kubernetes.io/projected/1e459f08-3f93-48bd-bb45-b950f416c87f-kube-api-access-7b89l\") pod \"multus-additional-cni-plugins-njk8v\" (UID: \"1e459f08-3f93-48bd-bb45-b950f416c87f\") " pod="openshift-multus/multus-additional-cni-plugins-njk8v"
Jul 28 17:09:14.255957 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254735    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-etc-modprobe-d\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.255957 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-etc-kubernetes\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.255957 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254793    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c643884f-420e-452c-98b0-09aede5dd675-socket-dir\") pod \"aws-ebs-csi-driver-node-t82zg\" (UID: \"c643884f-420e-452c-98b0-09aede5dd675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg"
Jul 28 17:09:14.255957 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254796    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1e459f08-3f93-48bd-bb45-b950f416c87f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-njk8v\" (UID: \"1e459f08-3f93-48bd-bb45-b950f416c87f\") " pod="openshift-multus/multus-additional-cni-plugins-njk8v"
Jul 28 17:09:14.255957 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254864    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-sys\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.255957 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254898    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-var-lib-kubelet\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.255957 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254910    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-etc-kubernetes\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.255957 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254924    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.255957 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254975    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-etc-sysctl-d\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.255957 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254983    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-sys\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.255957 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.254987    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-var-lib-kubelet\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.255957 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.255001    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b0a6ecd5-1b96-4003-ba79-fe147a67c757-ovnkube-config\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.255957 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.255037    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3b4343ba-84d3-4d95-8a25-e97b6cf88830-agent-certs\") pod \"konnectivity-agent-kc9pw\" (UID: \"3b4343ba-84d3-4d95-8a25-e97b6cf88830\") " pod="kube-system/konnectivity-agent-kc9pw"
Jul 28 17:09:14.256618 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.255055    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ed21aa75-f994-4548-93f7-99c172998ea1-etc-sysctl-d\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.256618 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.255063    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d4cfcb5-839a-431f-946c-6b5ac4e6c61e-host\") pod \"node-ca-ffvhq\" (UID: \"6d4cfcb5-839a-431f-946c-6b5ac4e6c61e\") " pod="openshift-image-registry/node-ca-ffvhq"
Jul 28 17:09:14.256618 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.255082    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/189dbf08-647d-42a8-975f-39838ad4edf9-dbus\") pod \"global-pull-secret-syncer-jwmpm\" (UID: \"189dbf08-647d-42a8-975f-39838ad4edf9\") " pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:14.256618 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.255122    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6d4cfcb5-839a-431f-946c-6b5ac4e6c61e-host\") pod \"node-ca-ffvhq\" (UID: \"6d4cfcb5-839a-431f-946c-6b5ac4e6c61e\") " pod="openshift-image-registry/node-ca-ffvhq"
Jul 28 17:09:14.256618 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.255251    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/189dbf08-647d-42a8-975f-39838ad4edf9-dbus\") pod \"global-pull-secret-syncer-jwmpm\" (UID: \"189dbf08-647d-42a8-975f-39838ad4edf9\") " pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:14.256871 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.256829    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed21aa75-f994-4548-93f7-99c172998ea1-tmp\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.256871 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.256862    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ed21aa75-f994-4548-93f7-99c172998ea1-etc-tuned\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.258824 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:14.258805    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:09:14.258824 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:14.258826    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:09:14.259005 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:14.258838    2581 projected.go:196] Error preparing data for projected volume kube-api-access-jgzn7 for pod openshift-network-diagnostics/network-check-target-2nt29: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:14.259005 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:14.258889    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33068506-3820-4d64-bc6b-33ea180dca70-kube-api-access-jgzn7 podName:33068506-3820-4d64-bc6b-33ea180dca70 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:14.758874888 +0000 UTC m=+3.033491313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jgzn7" (UniqueName: "kubernetes.io/projected/33068506-3820-4d64-bc6b-33ea180dca70-kube-api-access-jgzn7") pod "network-check-target-2nt29" (UID: "33068506-3820-4d64-bc6b-33ea180dca70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:14.261764 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.261701    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfjsz\" (UniqueName: \"kubernetes.io/projected/c643884f-420e-452c-98b0-09aede5dd675-kube-api-access-lfjsz\") pod \"aws-ebs-csi-driver-node-t82zg\" (UID: \"c643884f-420e-452c-98b0-09aede5dd675\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg"
Jul 28 17:09:14.261764 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.261716    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2bvc\" (UniqueName: \"kubernetes.io/projected/601b853c-8800-428b-89f6-1cb7b378600c-kube-api-access-r2bvc\") pod \"node-resolver-dbnhf\" (UID: \"601b853c-8800-428b-89f6-1cb7b378600c\") " pod="openshift-dns/node-resolver-dbnhf"
Jul 28 17:09:14.263001 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.262962    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsw86\" (UniqueName: \"kubernetes.io/projected/ed21aa75-f994-4548-93f7-99c172998ea1-kube-api-access-jsw86\") pod \"tuned-jhs8c\" (UID: \"ed21aa75-f994-4548-93f7-99c172998ea1\") " pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.263257 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.263200    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nrfd\" (UniqueName: \"kubernetes.io/projected/9a7b47e4-993e-4527-a00f-4ebbbc050348-kube-api-access-5nrfd\") pod \"network-metrics-daemon-zkf8x\" (UID: \"9a7b47e4-993e-4527-a00f-4ebbbc050348\") " pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:14.263342 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.263325    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fj75\" (UniqueName: \"kubernetes.io/projected/8188b00b-a96c-4bd3-a9b9-35fc129184d5-kube-api-access-8fj75\") pod \"iptables-alerter-pp6xv\" (UID: \"8188b00b-a96c-4bd3-a9b9-35fc129184d5\") " pod="openshift-network-operator/iptables-alerter-pp6xv"
Jul 28 17:09:14.265336 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.265305    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svdn6\" (UniqueName: \"kubernetes.io/projected/6d4cfcb5-839a-431f-946c-6b5ac4e6c61e-kube-api-access-svdn6\") pod \"node-ca-ffvhq\" (UID: \"6d4cfcb5-839a-431f-946c-6b5ac4e6c61e\") " pod="openshift-image-registry/node-ca-ffvhq"
Jul 28 17:09:14.267851 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.267820    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b89l\" (UniqueName: \"kubernetes.io/projected/1e459f08-3f93-48bd-bb45-b950f416c87f-kube-api-access-7b89l\") pod \"multus-additional-cni-plugins-njk8v\" (UID: \"1e459f08-3f93-48bd-bb45-b950f416c87f\") " pod="openshift-multus/multus-additional-cni-plugins-njk8v"
Jul 28 17:09:14.306447 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.306393    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-whdpj"
Jul 28 17:09:14.318014 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.317991    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-whdpj"
Jul 28 17:09:14.355736 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.355708    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b0a6ecd5-1b96-4003-ba79-fe147a67c757-ovnkube-config\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.355890 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.355746    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3b4343ba-84d3-4d95-8a25-e97b6cf88830-agent-certs\") pod \"konnectivity-agent-kc9pw\" (UID: \"3b4343ba-84d3-4d95-8a25-e97b6cf88830\") " pod="kube-system/konnectivity-agent-kc9pw"
Jul 28 17:09:14.355890 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.355774    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b0a6ecd5-1b96-4003-ba79-fe147a67c757-ovnkube-script-lib\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.355890 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.355799    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5a4cb725-44da-4042-8bc6-8f09251a6325-multus-daemon-config\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.355890 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.355822    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-etc-openvswitch\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.355890 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.355847    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-host-kubelet\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.356128 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.355921    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-etc-openvswitch\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.356128 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.355998    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-host-kubelet\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.356128 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.356115    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-host-slash\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.356414 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.356392    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5a4cb725-44da-4042-8bc6-8f09251a6325-multus-daemon-config\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.356480 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.356392    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b0a6ecd5-1b96-4003-ba79-fe147a67c757-ovnkube-config\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.356480 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.356448    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b0a6ecd5-1b96-4003-ba79-fe147a67c757-ovnkube-script-lib\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.356480 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.355868    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-host-slash\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.356583 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.356511    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-var-lib-openvswitch\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.356583 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.356538    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-system-cni-dir\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.356583 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.356565    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-host-run-netns\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.356719 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.356587    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b0a6ecd5-1b96-4003-ba79-fe147a67c757-env-overrides\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.356719 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.356614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-run-systemd\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.356719 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.356615    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-var-lib-openvswitch\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.356719 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.356630    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-system-cni-dir\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.356719 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.356636    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-host-run-ovn-kubernetes\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.356719 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.356620    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-host-run-netns\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.356719 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.356672    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-run-systemd\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.356719 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.356685    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-host-run-ovn-kubernetes\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.356719 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.356720    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-host-cni-netd\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.357166 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.356746    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnhp8\" (UniqueName: \"kubernetes.io/projected/b0a6ecd5-1b96-4003-ba79-fe147a67c757-kube-api-access-mnhp8\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.357166 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.356767    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-host-cni-netd\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.357166 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.356774    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-host-var-lib-cni-multus\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.357166 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.356820    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-host-var-lib-cni-multus\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.357166 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.356921    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-systemd-units\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.357166 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.356969    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5a4cb725-44da-4042-8bc6-8f09251a6325-cni-binary-copy\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.357166 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.356981    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-systemd-units\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.357166 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.356991    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b0a6ecd5-1b96-4003-ba79-fe147a67c757-env-overrides\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.357166 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.356997    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8md76\" (UniqueName: \"kubernetes.io/projected/5a4cb725-44da-4042-8bc6-8f09251a6325-kube-api-access-8md76\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.357166 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-node-log\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.357166 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357069    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-host-cni-bin\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.357166 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357091    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-multus-cni-dir\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.357166 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357117    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-cnibin\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.357166 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357139    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-multus-socket-dir-parent\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.357166 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357162    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-host-run-k8s-cni-cncf-io\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.357658 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357184    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-host-var-lib-kubelet\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.357658 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357197    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-node-log\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.357658 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357204    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-etc-kubernetes\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.357658 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357161    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-host-cni-bin\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.357658 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357230    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3b4343ba-84d3-4d95-8a25-e97b6cf88830-konnectivity-ca\") pod \"konnectivity-agent-kc9pw\" (UID: \"3b4343ba-84d3-4d95-8a25-e97b6cf88830\") " pod="kube-system/konnectivity-agent-kc9pw"
Jul 28 17:09:14.357658 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357252    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-multus-socket-dir-parent\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.357658 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357286    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-host-run-k8s-cni-cncf-io\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.357658 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357310    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-host-var-lib-kubelet\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.357658 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357346    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-run-ovn\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.357658 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357363    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-cnibin\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.357658 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357371    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0a6ecd5-1b96-4003-ba79-fe147a67c757-ovn-node-metrics-cert\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.357658 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357396    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5a4cb725-44da-4042-8bc6-8f09251a6325-cni-binary-copy\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.357658 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357424    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-multus-cni-dir\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.357658 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357424    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-multus-conf-dir\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.357658 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357454    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-multus-conf-dir\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.357658 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357483    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-log-socket\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.357658 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357522    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-etc-kubernetes\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.357658 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357548    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-os-release\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.358501 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357554    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-log-socket\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.358501 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357579    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-run-openvswitch\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.358501 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357594    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-run-ovn\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.358501 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357641    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-run-openvswitch\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.358501 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357695    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-os-release\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.358501 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357720    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-host-run-netns\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.358501 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357748    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-hostroot\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.358501 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357776    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-host-run-multus-certs\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.358501 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357781    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-hostroot\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.358501 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357777    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3b4343ba-84d3-4d95-8a25-e97b6cf88830-konnectivity-ca\") pod \"konnectivity-agent-kc9pw\" (UID: \"3b4343ba-84d3-4d95-8a25-e97b6cf88830\") " pod="kube-system/konnectivity-agent-kc9pw"
Jul 28 17:09:14.358501 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357801    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-host-var-lib-cni-bin\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.358501 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357813    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-host-run-multus-certs\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.358501 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357840    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.358501 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357849    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-host-var-lib-cni-bin\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.358501 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357915    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0a6ecd5-1b96-4003-ba79-fe147a67c757-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.358501 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.357990    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a4cb725-44da-4042-8bc6-8f09251a6325-host-run-netns\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.359309 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.359012    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3b4343ba-84d3-4d95-8a25-e97b6cf88830-agent-certs\") pod \"konnectivity-agent-kc9pw\" (UID: \"3b4343ba-84d3-4d95-8a25-e97b6cf88830\") " pod="kube-system/konnectivity-agent-kc9pw"
Jul 28 17:09:14.359790 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.359765    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0a6ecd5-1b96-4003-ba79-fe147a67c757-ovn-node-metrics-cert\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.366169 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.366148    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8md76\" (UniqueName: \"kubernetes.io/projected/5a4cb725-44da-4042-8bc6-8f09251a6325-kube-api-access-8md76\") pod \"multus-r5x5j\" (UID: \"5a4cb725-44da-4042-8bc6-8f09251a6325\") " pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.366252 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.366180    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnhp8\" (UniqueName: \"kubernetes.io/projected/b0a6ecd5-1b96-4003-ba79-fe147a67c757-kube-api-access-mnhp8\") pod \"ovnkube-node-9l7dh\" (UID: \"b0a6ecd5-1b96-4003-ba79-fe147a67c757\") " pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.434776 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.434738    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-njk8v"
Jul 28 17:09:14.443331 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.443310    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg"
Jul 28 17:09:14.452776 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.452758    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dbnhf"
Jul 28 17:09:14.459362 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.459342    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ffvhq"
Jul 28 17:09:14.466721 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.466704    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jhs8c"
Jul 28 17:09:14.474200 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.474184    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pp6xv"
Jul 28 17:09:14.479766 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.479746    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:14.485890 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.485873    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kc9pw"
Jul 28 17:09:14.491457 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.491441    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r5x5j"
Jul 28 17:09:14.760078 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.760015    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgzn7\" (UniqueName: \"kubernetes.io/projected/33068506-3820-4d64-bc6b-33ea180dca70-kube-api-access-jgzn7\") pod \"network-check-target-2nt29\" (UID: \"33068506-3820-4d64-bc6b-33ea180dca70\") " pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:09:14.760078 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.760053    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a7b47e4-993e-4527-a00f-4ebbbc050348-metrics-certs\") pod \"network-metrics-daemon-zkf8x\" (UID: \"9a7b47e4-993e-4527-a00f-4ebbbc050348\") " pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:14.760236 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:14.760081    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/189dbf08-647d-42a8-975f-39838ad4edf9-original-pull-secret\") pod \"global-pull-secret-syncer-jwmpm\" (UID: \"189dbf08-647d-42a8-975f-39838ad4edf9\") " pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:14.760236 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:14.760170    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:14.760236 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:14.760227    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a7b47e4-993e-4527-a00f-4ebbbc050348-metrics-certs podName:9a7b47e4-993e-4527-a00f-4ebbbc050348 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:15.760213095 +0000 UTC m=+4.034829509 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a7b47e4-993e-4527-a00f-4ebbbc050348-metrics-certs") pod "network-metrics-daemon-zkf8x" (UID: "9a7b47e4-993e-4527-a00f-4ebbbc050348") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:14.760342 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:14.760171    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:09:14.760342 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:14.760274    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:09:14.760342 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:14.760285    2581 projected.go:196] Error preparing data for projected volume kube-api-access-jgzn7 for pod openshift-network-diagnostics/network-check-target-2nt29: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:14.760342 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:14.760170    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:14.760457 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:14.760330    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33068506-3820-4d64-bc6b-33ea180dca70-kube-api-access-jgzn7 podName:33068506-3820-4d64-bc6b-33ea180dca70 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:15.760311667 +0000 UTC m=+4.034928091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgzn7" (UniqueName: "kubernetes.io/projected/33068506-3820-4d64-bc6b-33ea180dca70-kube-api-access-jgzn7") pod "network-check-target-2nt29" (UID: "33068506-3820-4d64-bc6b-33ea180dca70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:14.760457 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:14.760413    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/189dbf08-647d-42a8-975f-39838ad4edf9-original-pull-secret podName:189dbf08-647d-42a8-975f-39838ad4edf9 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:15.760400125 +0000 UTC m=+4.035016535 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/189dbf08-647d-42a8-975f-39838ad4edf9-original-pull-secret") pod "global-pull-secret-syncer-jwmpm" (UID: "189dbf08-647d-42a8-975f-39838ad4edf9") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:15.072419 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:15.072351    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:09:15.101301 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:15.101274    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4cb725_44da_4042_8bc6_8f09251a6325.slice/crio-00d0a7f95c1206836414c5d8b2623167cd03f79f45e9efe4605e75aa242de044 WatchSource:0}: Error finding container 00d0a7f95c1206836414c5d8b2623167cd03f79f45e9efe4605e75aa242de044: Status 404 returned error can't find the container with id 00d0a7f95c1206836414c5d8b2623167cd03f79f45e9efe4605e75aa242de044
Jul 28 17:09:15.102527 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:15.102503    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc643884f_420e_452c_98b0_09aede5dd675.slice/crio-702ead8b23dabf44a8dbcb02f43c92f91ded4ca3c70361069ab0f0d16d39a9ca WatchSource:0}: Error finding container 702ead8b23dabf44a8dbcb02f43c92f91ded4ca3c70361069ab0f0d16d39a9ca: Status 404 returned error can't find the container with id 702ead8b23dabf44a8dbcb02f43c92f91ded4ca3c70361069ab0f0d16d39a9ca
Jul 28 17:09:15.104544 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:15.103570    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a6ecd5_1b96_4003_ba79_fe147a67c757.slice/crio-17d00b578acc14f40b1b8b7dc408e8ffc01c561fcd6e7245009814fc704ce786 WatchSource:0}: Error finding container 17d00b578acc14f40b1b8b7dc408e8ffc01c561fcd6e7245009814fc704ce786: Status 404 returned error can't find the container with id 17d00b578acc14f40b1b8b7dc408e8ffc01c561fcd6e7245009814fc704ce786
Jul 28 17:09:15.104860 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:15.104840    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4343ba_84d3_4d95_8a25_e97b6cf88830.slice/crio-d4440e51ac8d29fcd95267ff545afa1c8bef48b56acdc2b731ac82911098ca4e WatchSource:0}: Error finding container d4440e51ac8d29fcd95267ff545afa1c8bef48b56acdc2b731ac82911098ca4e: Status 404 returned error can't find the container with id d4440e51ac8d29fcd95267ff545afa1c8bef48b56acdc2b731ac82911098ca4e
Jul 28 17:09:15.126375 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:15.126313    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded21aa75_f994_4548_93f7_99c172998ea1.slice/crio-d31b9b255956b3821eec8a9575b12a8e0f5080d256ea876afaf640564f052e6c WatchSource:0}: Error finding container d31b9b255956b3821eec8a9575b12a8e0f5080d256ea876afaf640564f052e6c: Status 404 returned error can't find the container with id d31b9b255956b3821eec8a9575b12a8e0f5080d256ea876afaf640564f052e6c
Jul 28 17:09:15.128688 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:15.128665    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d4cfcb5_839a_431f_946c_6b5ac4e6c61e.slice/crio-c54721914edb5c513cb77ab83b11e7cc7f00edbef98e30a63b95d7589f26b353 WatchSource:0}: Error finding container c54721914edb5c513cb77ab83b11e7cc7f00edbef98e30a63b95d7589f26b353: Status 404 returned error can't find the container with id c54721914edb5c513cb77ab83b11e7cc7f00edbef98e30a63b95d7589f26b353
Jul 28 17:09:15.130570 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:15.130543    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e459f08_3f93_48bd_bb45_b950f416c87f.slice/crio-89ff1285637a923b7eeb5d37a9f616be8d33e224eeabde9d7cac5c06357c08c1 WatchSource:0}: Error finding container 89ff1285637a923b7eeb5d37a9f616be8d33e224eeabde9d7cac5c06357c08c1: Status 404 returned error can't find the container with id 89ff1285637a923b7eeb5d37a9f616be8d33e224eeabde9d7cac5c06357c08c1
Jul 28 17:09:15.130895 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:15.130818    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8188b00b_a96c_4bd3_a9b9_35fc129184d5.slice/crio-159485a17ee3ca52fb21fc3650ebe870525bf27ad30363b1dd21488175ea1717 WatchSource:0}: Error finding container 159485a17ee3ca52fb21fc3650ebe870525bf27ad30363b1dd21488175ea1717: Status 404 returned error can't find the container with id 159485a17ee3ca52fb21fc3650ebe870525bf27ad30363b1dd21488175ea1717
Jul 28 17:09:15.133103 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:15.132862    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod601b853c_8800_428b_89f6_1cb7b378600c.slice/crio-cd9241ea4b67cde3e1aa63c4ba2763e41de7cf7b02377dfdb089e6a00446c4fc WatchSource:0}: Error finding container cd9241ea4b67cde3e1aa63c4ba2763e41de7cf7b02377dfdb089e6a00446c4fc: Status 404 returned error can't find the container with id cd9241ea4b67cde3e1aa63c4ba2763e41de7cf7b02377dfdb089e6a00446c4fc
Jul 28 17:09:15.293351 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:15.293164    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r5x5j" event={"ID":"5a4cb725-44da-4042-8bc6-8f09251a6325","Type":"ContainerStarted","Data":"00d0a7f95c1206836414c5d8b2623167cd03f79f45e9efe4605e75aa242de044"}
Jul 28 17:09:15.294205 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:15.294182    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dbnhf" event={"ID":"601b853c-8800-428b-89f6-1cb7b378600c","Type":"ContainerStarted","Data":"cd9241ea4b67cde3e1aa63c4ba2763e41de7cf7b02377dfdb089e6a00446c4fc"}
Jul 28 17:09:15.295266 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:15.295238    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njk8v" event={"ID":"1e459f08-3f93-48bd-bb45-b950f416c87f","Type":"ContainerStarted","Data":"89ff1285637a923b7eeb5d37a9f616be8d33e224eeabde9d7cac5c06357c08c1"}
Jul 28 17:09:15.297825 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:15.297799    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jhs8c" event={"ID":"ed21aa75-f994-4548-93f7-99c172998ea1","Type":"ContainerStarted","Data":"d31b9b255956b3821eec8a9575b12a8e0f5080d256ea876afaf640564f052e6c"}
Jul 28 17:09:15.299071 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:15.299040    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kc9pw" event={"ID":"3b4343ba-84d3-4d95-8a25-e97b6cf88830","Type":"ContainerStarted","Data":"d4440e51ac8d29fcd95267ff545afa1c8bef48b56acdc2b731ac82911098ca4e"}
Jul 28 17:09:15.300069 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:15.300049    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh" event={"ID":"b0a6ecd5-1b96-4003-ba79-fe147a67c757","Type":"ContainerStarted","Data":"17d00b578acc14f40b1b8b7dc408e8ffc01c561fcd6e7245009814fc704ce786"}
Jul 28 17:09:15.301178 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:15.301157    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg" event={"ID":"c643884f-420e-452c-98b0-09aede5dd675","Type":"ContainerStarted","Data":"702ead8b23dabf44a8dbcb02f43c92f91ded4ca3c70361069ab0f0d16d39a9ca"}
Jul 28 17:09:15.303003 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:15.302976    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-231.ec2.internal" event={"ID":"83f9b4a12578fa1633b19325b5319dc4","Type":"ContainerStarted","Data":"9ae63c9fcbbfc061431fbbad8ca75c38c899c26106b4c4f9a7e994a889bd7ef6"}
Jul 28 17:09:15.304083 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:15.304057    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pp6xv" event={"ID":"8188b00b-a96c-4bd3-a9b9-35fc129184d5","Type":"ContainerStarted","Data":"159485a17ee3ca52fb21fc3650ebe870525bf27ad30363b1dd21488175ea1717"}
Jul 28 17:09:15.305037 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:15.305017    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ffvhq" event={"ID":"6d4cfcb5-839a-431f-946c-6b5ac4e6c61e","Type":"ContainerStarted","Data":"c54721914edb5c513cb77ab83b11e7cc7f00edbef98e30a63b95d7589f26b353"}
Jul 28 17:09:15.320022 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:15.319984    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-231.ec2.internal" podStartSLOduration=2.319972451 podStartE2EDuration="2.319972451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:15.319650246 +0000 UTC m=+3.594266678" watchObservedRunningTime="2026-07-28 17:09:15.319972451 +0000 UTC m=+3.594588882"
Jul 28 17:09:15.322786 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:15.322720    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:04:14 +0000 UTC" deadline="2028-05-03 21:45:07.316501726 +0000 UTC"
Jul 28 17:09:15.322786 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:15.322743    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15484h35m51.993762789s"
Jul 28 17:09:15.769184 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:15.768797    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a7b47e4-993e-4527-a00f-4ebbbc050348-metrics-certs\") pod \"network-metrics-daemon-zkf8x\" (UID: \"9a7b47e4-993e-4527-a00f-4ebbbc050348\") " pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:15.769184 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:15.768852    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/189dbf08-647d-42a8-975f-39838ad4edf9-original-pull-secret\") pod \"global-pull-secret-syncer-jwmpm\" (UID: \"189dbf08-647d-42a8-975f-39838ad4edf9\") " pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:15.769184 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:15.768913    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgzn7\" (UniqueName: \"kubernetes.io/projected/33068506-3820-4d64-bc6b-33ea180dca70-kube-api-access-jgzn7\") pod \"network-check-target-2nt29\" (UID: \"33068506-3820-4d64-bc6b-33ea180dca70\") " pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:09:15.769184 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:15.768994    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:15.769184 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:15.769014    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:15.769184 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:15.769032    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:09:15.769184 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:15.769049    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:09:15.769184 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:15.769061    2581 projected.go:196] Error preparing data for projected volume kube-api-access-jgzn7 for pod openshift-network-diagnostics/network-check-target-2nt29: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:15.769184 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:15.769065    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a7b47e4-993e-4527-a00f-4ebbbc050348-metrics-certs podName:9a7b47e4-993e-4527-a00f-4ebbbc050348 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:17.769044824 +0000 UTC m=+6.043661255 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a7b47e4-993e-4527-a00f-4ebbbc050348-metrics-certs") pod "network-metrics-daemon-zkf8x" (UID: "9a7b47e4-993e-4527-a00f-4ebbbc050348") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:15.769184 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:15.769084    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/189dbf08-647d-42a8-975f-39838ad4edf9-original-pull-secret podName:189dbf08-647d-42a8-975f-39838ad4edf9 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:17.769074447 +0000 UTC m=+6.043690864 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/189dbf08-647d-42a8-975f-39838ad4edf9-original-pull-secret") pod "global-pull-secret-syncer-jwmpm" (UID: "189dbf08-647d-42a8-975f-39838ad4edf9") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:15.769184 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:15.769114    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33068506-3820-4d64-bc6b-33ea180dca70-kube-api-access-jgzn7 podName:33068506-3820-4d64-bc6b-33ea180dca70 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:17.769097555 +0000 UTC m=+6.043713968 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgzn7" (UniqueName: "kubernetes.io/projected/33068506-3820-4d64-bc6b-33ea180dca70-kube-api-access-jgzn7") pod "network-check-target-2nt29" (UID: "33068506-3820-4d64-bc6b-33ea180dca70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:16.288498 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:16.287686    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:16.288498 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:16.287814    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwmpm" podUID="189dbf08-647d-42a8-975f-39838ad4edf9"
Jul 28 17:09:16.288498 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:16.288205    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:09:16.288498 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:16.288293    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nt29" podUID="33068506-3820-4d64-bc6b-33ea180dca70"
Jul 28 17:09:16.288498 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:16.288367    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:16.288498 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:16.288450    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkf8x" podUID="9a7b47e4-993e-4527-a00f-4ebbbc050348"
Jul 28 17:09:16.323222 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:16.323173    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:04:14 +0000 UTC" deadline="2028-01-07 18:10:52.646767186 +0000 UTC"
Jul 28 17:09:16.323222 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:16.323218    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12673h1m36.323553754s"
Jul 28 17:09:16.328304 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:16.327117    2581 generic.go:356] "Generic (PLEG): container finished" podID="d6006a44a5831824d4f450fa2922581d" containerID="84d0dbd8a61a098a4c2e36c19b8c10d81da013a8128059c5afca5711cf2663bb" exitCode=0
Jul 28 17:09:16.328304 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:16.328064    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal" event={"ID":"d6006a44a5831824d4f450fa2922581d","Type":"ContainerDied","Data":"84d0dbd8a61a098a4c2e36c19b8c10d81da013a8128059c5afca5711cf2663bb"}
Jul 28 17:09:17.348173 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:17.348100    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal" event={"ID":"d6006a44a5831824d4f450fa2922581d","Type":"ContainerStarted","Data":"80886952c6fc3d5b06ef849cfe20b4034d4d05d5997a5991b812c3f2e5320a78"}
Jul 28 17:09:17.792793 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:17.788298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgzn7\" (UniqueName: \"kubernetes.io/projected/33068506-3820-4d64-bc6b-33ea180dca70-kube-api-access-jgzn7\") pod \"network-check-target-2nt29\" (UID: \"33068506-3820-4d64-bc6b-33ea180dca70\") " pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:09:17.792793 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:17.788363    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a7b47e4-993e-4527-a00f-4ebbbc050348-metrics-certs\") pod \"network-metrics-daemon-zkf8x\" (UID: \"9a7b47e4-993e-4527-a00f-4ebbbc050348\") " pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:17.792793 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:17.788401    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/189dbf08-647d-42a8-975f-39838ad4edf9-original-pull-secret\") pod \"global-pull-secret-syncer-jwmpm\" (UID: \"189dbf08-647d-42a8-975f-39838ad4edf9\") " pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:17.792793 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:17.788517    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:17.792793 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:17.788576    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/189dbf08-647d-42a8-975f-39838ad4edf9-original-pull-secret podName:189dbf08-647d-42a8-975f-39838ad4edf9 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:21.788556759 +0000 UTC m=+10.063173185 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/189dbf08-647d-42a8-975f-39838ad4edf9-original-pull-secret") pod "global-pull-secret-syncer-jwmpm" (UID: "189dbf08-647d-42a8-975f-39838ad4edf9") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:17.792793 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:17.792464    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:17.792793 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:17.792509    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:09:17.792793 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:17.792523    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a7b47e4-993e-4527-a00f-4ebbbc050348-metrics-certs podName:9a7b47e4-993e-4527-a00f-4ebbbc050348 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:21.792511018 +0000 UTC m=+10.067127444 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a7b47e4-993e-4527-a00f-4ebbbc050348-metrics-certs") pod "network-metrics-daemon-zkf8x" (UID: "9a7b47e4-993e-4527-a00f-4ebbbc050348") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:17.792793 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:17.792532    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:09:17.792793 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:17.792548    2581 projected.go:196] Error preparing data for projected volume kube-api-access-jgzn7 for pod openshift-network-diagnostics/network-check-target-2nt29: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:17.792793 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:17.792598    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33068506-3820-4d64-bc6b-33ea180dca70-kube-api-access-jgzn7 podName:33068506-3820-4d64-bc6b-33ea180dca70 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:21.792582017 +0000 UTC m=+10.067198428 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgzn7" (UniqueName: "kubernetes.io/projected/33068506-3820-4d64-bc6b-33ea180dca70-kube-api-access-jgzn7") pod "network-check-target-2nt29" (UID: "33068506-3820-4d64-bc6b-33ea180dca70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:18.285829 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:18.285755    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:18.286505 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:18.285982    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkf8x" podUID="9a7b47e4-993e-4527-a00f-4ebbbc050348"
Jul 28 17:09:18.286505 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:18.286300    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:09:18.286505 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:18.286380    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nt29" podUID="33068506-3820-4d64-bc6b-33ea180dca70"
Jul 28 17:09:18.286505 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:18.286410    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:18.286505 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:18.286467    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwmpm" podUID="189dbf08-647d-42a8-975f-39838ad4edf9"
Jul 28 17:09:20.285690 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:20.285658    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:09:20.286148 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:20.285775    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nt29" podUID="33068506-3820-4d64-bc6b-33ea180dca70"
Jul 28 17:09:20.286246 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:20.286229    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:20.286362 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:20.286343    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkf8x" podUID="9a7b47e4-993e-4527-a00f-4ebbbc050348"
Jul 28 17:09:20.286449 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:20.286436    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:20.286546 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:20.286521    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwmpm" podUID="189dbf08-647d-42a8-975f-39838ad4edf9"
Jul 28 17:09:21.824673 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:21.824542    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgzn7\" (UniqueName: \"kubernetes.io/projected/33068506-3820-4d64-bc6b-33ea180dca70-kube-api-access-jgzn7\") pod \"network-check-target-2nt29\" (UID: \"33068506-3820-4d64-bc6b-33ea180dca70\") " pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:09:21.824673 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:21.824605    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a7b47e4-993e-4527-a00f-4ebbbc050348-metrics-certs\") pod \"network-metrics-daemon-zkf8x\" (UID: \"9a7b47e4-993e-4527-a00f-4ebbbc050348\") " pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:21.824673 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:21.824645    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/189dbf08-647d-42a8-975f-39838ad4edf9-original-pull-secret\") pod \"global-pull-secret-syncer-jwmpm\" (UID: \"189dbf08-647d-42a8-975f-39838ad4edf9\") " pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:21.825231 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:21.824713    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:09:21.825231 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:21.824735    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:09:21.825231 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:21.824748    2581 projected.go:196] Error preparing data for projected volume kube-api-access-jgzn7 for pod openshift-network-diagnostics/network-check-target-2nt29: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:21.825231 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:21.824778    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:21.825231 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:21.824810    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33068506-3820-4d64-bc6b-33ea180dca70-kube-api-access-jgzn7 podName:33068506-3820-4d64-bc6b-33ea180dca70 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:29.824792984 +0000 UTC m=+18.099409412 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgzn7" (UniqueName: "kubernetes.io/projected/33068506-3820-4d64-bc6b-33ea180dca70-kube-api-access-jgzn7") pod "network-check-target-2nt29" (UID: "33068506-3820-4d64-bc6b-33ea180dca70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:21.825231 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:21.824821    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:21.825231 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:21.824844    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a7b47e4-993e-4527-a00f-4ebbbc050348-metrics-certs podName:9a7b47e4-993e-4527-a00f-4ebbbc050348 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:29.82482609 +0000 UTC m=+18.099442500 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a7b47e4-993e-4527-a00f-4ebbbc050348-metrics-certs") pod "network-metrics-daemon-zkf8x" (UID: "9a7b47e4-993e-4527-a00f-4ebbbc050348") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:21.825231 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:21.824861    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/189dbf08-647d-42a8-975f-39838ad4edf9-original-pull-secret podName:189dbf08-647d-42a8-975f-39838ad4edf9 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:29.824851774 +0000 UTC m=+18.099468184 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/189dbf08-647d-42a8-975f-39838ad4edf9-original-pull-secret") pod "global-pull-secret-syncer-jwmpm" (UID: "189dbf08-647d-42a8-975f-39838ad4edf9") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:22.286022 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:22.285819    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:22.286022 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:22.285921    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:22.286022 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:22.285936    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkf8x" podUID="9a7b47e4-993e-4527-a00f-4ebbbc050348"
Jul 28 17:09:22.286291 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:22.286052    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwmpm" podUID="189dbf08-647d-42a8-975f-39838ad4edf9"
Jul 28 17:09:22.286291 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:22.286099    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:09:22.286291 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:22.286162    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nt29" podUID="33068506-3820-4d64-bc6b-33ea180dca70"
Jul 28 17:09:24.285341 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:24.285303    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:24.285788 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:24.285438    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwmpm" podUID="189dbf08-647d-42a8-975f-39838ad4edf9"
Jul 28 17:09:24.285880 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:24.285857    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:09:24.285934 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:24.285892    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:24.286009 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:24.285985    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nt29" podUID="33068506-3820-4d64-bc6b-33ea180dca70"
Jul 28 17:09:24.286103 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:24.286084    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkf8x" podUID="9a7b47e4-993e-4527-a00f-4ebbbc050348"
Jul 28 17:09:26.285108 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:26.285032    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:26.285544 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:26.285173    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkf8x" podUID="9a7b47e4-993e-4527-a00f-4ebbbc050348"
Jul 28 17:09:26.285544 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:26.285213    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:26.285544 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:26.285294    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwmpm" podUID="189dbf08-647d-42a8-975f-39838ad4edf9"
Jul 28 17:09:26.285544 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:26.285335    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:09:26.285544 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:26.285402    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nt29" podUID="33068506-3820-4d64-bc6b-33ea180dca70"
Jul 28 17:09:28.284958 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:28.284915    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:28.285483 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:28.284915    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:09:28.285483 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:28.285068    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:28.285483 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:28.285071    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwmpm" podUID="189dbf08-647d-42a8-975f-39838ad4edf9"
Jul 28 17:09:28.285483 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:28.285144    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nt29" podUID="33068506-3820-4d64-bc6b-33ea180dca70"
Jul 28 17:09:28.285483 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:28.285258    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkf8x" podUID="9a7b47e4-993e-4527-a00f-4ebbbc050348"
Jul 28 17:09:29.880386 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:29.880347    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/189dbf08-647d-42a8-975f-39838ad4edf9-original-pull-secret\") pod \"global-pull-secret-syncer-jwmpm\" (UID: \"189dbf08-647d-42a8-975f-39838ad4edf9\") " pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:29.880932 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:29.880506    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgzn7\" (UniqueName: \"kubernetes.io/projected/33068506-3820-4d64-bc6b-33ea180dca70-kube-api-access-jgzn7\") pod \"network-check-target-2nt29\" (UID: \"33068506-3820-4d64-bc6b-33ea180dca70\") " pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:09:29.880932 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:29.880511    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:29.880932 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:29.880545    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a7b47e4-993e-4527-a00f-4ebbbc050348-metrics-certs\") pod \"network-metrics-daemon-zkf8x\" (UID: \"9a7b47e4-993e-4527-a00f-4ebbbc050348\") " pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:29.880932 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:29.880584    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/189dbf08-647d-42a8-975f-39838ad4edf9-original-pull-secret podName:189dbf08-647d-42a8-975f-39838ad4edf9 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:45.88056431 +0000 UTC m=+34.155180740 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/189dbf08-647d-42a8-975f-39838ad4edf9-original-pull-secret") pod "global-pull-secret-syncer-jwmpm" (UID: "189dbf08-647d-42a8-975f-39838ad4edf9") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:29.880932 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:29.880642    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:09:29.880932 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:29.880666    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:09:29.880932 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:29.880642    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:29.880932 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:29.880678    2581 projected.go:196] Error preparing data for projected volume kube-api-access-jgzn7 for pod openshift-network-diagnostics/network-check-target-2nt29: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:29.880932 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:29.880740    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a7b47e4-993e-4527-a00f-4ebbbc050348-metrics-certs podName:9a7b47e4-993e-4527-a00f-4ebbbc050348 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:45.88071911 +0000 UTC m=+34.155335533 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a7b47e4-993e-4527-a00f-4ebbbc050348-metrics-certs") pod "network-metrics-daemon-zkf8x" (UID: "9a7b47e4-993e-4527-a00f-4ebbbc050348") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:29.880932 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:29.880766    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33068506-3820-4d64-bc6b-33ea180dca70-kube-api-access-jgzn7 podName:33068506-3820-4d64-bc6b-33ea180dca70 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:45.88075464 +0000 UTC m=+34.155371059 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgzn7" (UniqueName: "kubernetes.io/projected/33068506-3820-4d64-bc6b-33ea180dca70-kube-api-access-jgzn7") pod "network-check-target-2nt29" (UID: "33068506-3820-4d64-bc6b-33ea180dca70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:30.284929 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:30.284857    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:30.285094 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:30.285000    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwmpm" podUID="189dbf08-647d-42a8-975f-39838ad4edf9"
Jul 28 17:09:30.285425 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:30.285228    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:09:30.285425 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:30.285319    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nt29" podUID="33068506-3820-4d64-bc6b-33ea180dca70"
Jul 28 17:09:30.285425 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:30.285235    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:30.285425 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:30.285401    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkf8x" podUID="9a7b47e4-993e-4527-a00f-4ebbbc050348"
Jul 28 17:09:32.286673 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:32.286630    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:09:32.287383 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:32.287354    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nt29" podUID="33068506-3820-4d64-bc6b-33ea180dca70"
Jul 28 17:09:32.287936 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:32.287914    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:32.288212 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:32.288192    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkf8x" podUID="9a7b47e4-993e-4527-a00f-4ebbbc050348"
Jul 28 17:09:32.288406 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:32.288392    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:32.288598 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:32.288581    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwmpm" podUID="189dbf08-647d-42a8-975f-39838ad4edf9"
Jul 28 17:09:34.285929 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:34.285735    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:34.286496 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:34.285736    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:09:34.286496 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:34.285993    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwmpm" podUID="189dbf08-647d-42a8-975f-39838ad4edf9"
Jul 28 17:09:34.286496 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:34.285747    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:34.286496 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:34.286039    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nt29" podUID="33068506-3820-4d64-bc6b-33ea180dca70"
Jul 28 17:09:34.286496 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:34.286152    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkf8x" podUID="9a7b47e4-993e-4527-a00f-4ebbbc050348"
Jul 28 17:09:34.380996 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:34.380971    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ffvhq" event={"ID":"6d4cfcb5-839a-431f-946c-6b5ac4e6c61e","Type":"ContainerStarted","Data":"30bd8c83525118d7744efbed1c1d7b07eb74fbc224b11e462f40d42d3d038fdf"}
Jul 28 17:09:34.383730 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:34.383705    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r5x5j" event={"ID":"5a4cb725-44da-4042-8bc6-8f09251a6325","Type":"ContainerStarted","Data":"967ee61839385dc14d31ac98590b3f8ff85582031def0c56a7c4ffb2dbe09d99"}
Jul 28 17:09:34.385256 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:34.385236    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dbnhf" event={"ID":"601b853c-8800-428b-89f6-1cb7b378600c","Type":"ContainerStarted","Data":"c72ff0f284c6488a6766f8510a201661e3bf3b6cb46e3c6e696b328a6576cef9"}
Jul 28 17:09:34.386539 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:34.386517    2581 generic.go:356] "Generic (PLEG): container finished" podID="1e459f08-3f93-48bd-bb45-b950f416c87f" containerID="e92a318125a255483abe4d33c417bd7e76ec0b25c8f22ff531bf0299a57a8e32" exitCode=0
Jul 28 17:09:34.386615 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:34.386584    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njk8v" event={"ID":"1e459f08-3f93-48bd-bb45-b950f416c87f","Type":"ContainerDied","Data":"e92a318125a255483abe4d33c417bd7e76ec0b25c8f22ff531bf0299a57a8e32"}
Jul 28 17:09:34.387981 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:34.387963    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jhs8c" event={"ID":"ed21aa75-f994-4548-93f7-99c172998ea1","Type":"ContainerStarted","Data":"029b0f0c444bca696c71f67480adfadc38fd26b13ca33c4d03df8e054ac2c571"}
Jul 28 17:09:34.389316 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:34.389297    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kc9pw" event={"ID":"3b4343ba-84d3-4d95-8a25-e97b6cf88830","Type":"ContainerStarted","Data":"25eda4bdaa4bc53c4be4300d8061a5d08d3530e60fad521e42731ee06354ef8e"}
Jul 28 17:09:34.391291 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:34.391276    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 17:09:34.391613 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:34.391575    2581 generic.go:356] "Generic (PLEG): container finished" podID="b0a6ecd5-1b96-4003-ba79-fe147a67c757" containerID="adc1996482b2b66affd4f664792f231db443e6be0c4a22eaaa4fc38ab1c7a7fe" exitCode=1
Jul 28 17:09:34.391693 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:34.391636    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh" event={"ID":"b0a6ecd5-1b96-4003-ba79-fe147a67c757","Type":"ContainerStarted","Data":"8cf42ff150ee8b0413bf1cd4a6d92c88e0dd902992ae96111563968cbd9eaa46"}
Jul 28 17:09:34.391742 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:34.391699    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh" event={"ID":"b0a6ecd5-1b96-4003-ba79-fe147a67c757","Type":"ContainerStarted","Data":"b821e8530237c418efea765bf7f1901121a58b72fb8c066c11cf56ad061af7b9"}
Jul 28 17:09:34.391782 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:34.391746    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh" event={"ID":"b0a6ecd5-1b96-4003-ba79-fe147a67c757","Type":"ContainerDied","Data":"adc1996482b2b66affd4f664792f231db443e6be0c4a22eaaa4fc38ab1c7a7fe"}
Jul 28 17:09:34.391828 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:34.391802    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh" event={"ID":"b0a6ecd5-1b96-4003-ba79-fe147a67c757","Type":"ContainerStarted","Data":"652d7da235f19e5997478652f597d49bb016dfda73be9561aa12018a0aabe7c1"}
Jul 28 17:09:34.393711 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:34.393686    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg" event={"ID":"c643884f-420e-452c-98b0-09aede5dd675","Type":"ContainerStarted","Data":"aefc63f4b169e5b717551778bda4137a9686fe43b9a54c51fc55f903bf11875c"}
Jul 28 17:09:34.413204 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:34.413171    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-231.ec2.internal" podStartSLOduration=21.413160631 podStartE2EDuration="21.413160631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:17.367005062 +0000 UTC m=+5.641621496" watchObservedRunningTime="2026-07-28 17:09:34.413160631 +0000 UTC m=+22.687777063"
Jul 28 17:09:34.431581 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:34.431536    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jhs8c" podStartSLOduration=3.708087439 podStartE2EDuration="22.431519113s" totalImagesPullingTime="18.723431674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:34.430832721 +0000 UTC m=+22.705449153" watchObservedRunningTime="2026-07-28 17:09:34.431519113 +0000 UTC m=+22.706135545"
Jul 28 17:09:34.454777 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:34.454742    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-r5x5j" podStartSLOduration=3.5882989199999997 podStartE2EDuration="22.454728582s" totalImagesPullingTime="18.866429662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:34.4544648 +0000 UTC m=+22.729081232" watchObservedRunningTime="2026-07-28 17:09:34.454728582 +0000 UTC m=+22.729345016"
Jul 28 17:09:34.474445 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:34.474397    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dbnhf" podStartSLOduration=3.822016534 podStartE2EDuration="22.474379036s" totalImagesPullingTime="18.652362502s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:34.474155673 +0000 UTC m=+22.748772100" watchObservedRunningTime="2026-07-28 17:09:34.474379036 +0000 UTC m=+22.748995470"
Jul 28 17:09:34.491170 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:34.491119    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ffvhq" podStartSLOduration=3.815667358 podStartE2EDuration="22.491102925s" totalImagesPullingTime="18.675435567s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:34.490984973 +0000 UTC m=+22.765601406" watchObservedRunningTime="2026-07-28 17:09:34.491102925 +0000 UTC m=+22.765719359"
Jul 28 17:09:34.507808 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:34.507763    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kc9pw" podStartSLOduration=3.845632707 podStartE2EDuration="22.507744251s" totalImagesPullingTime="18.662111544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:34.507244889 +0000 UTC m=+22.781861322" watchObservedRunningTime="2026-07-28 17:09:34.507744251 +0000 UTC m=+22.782360683"
Jul 28 17:09:35.178848 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:35.178825    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 17:09:35.221259 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:35.221155    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T17:09:35.17884439Z","UUID":"a89ac1a8-6679-4397-9079-f5877e7e6389","Handler":null,"Name":"","Endpoint":""}
Jul 28 17:09:35.222612 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:35.222594    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 17:09:35.222718 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:35.222628    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 17:09:35.397694 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:35.397603    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pp6xv" event={"ID":"8188b00b-a96c-4bd3-a9b9-35fc129184d5","Type":"ContainerStarted","Data":"b51f4b81de9c02138a19bf83d6ab3de3a891a0140643a61b9c70742a1263c91e"}
Jul 28 17:09:35.400820 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:35.400794    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 17:09:35.401319 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:35.401214    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh" event={"ID":"b0a6ecd5-1b96-4003-ba79-fe147a67c757","Type":"ContainerStarted","Data":"200af51d7de04133f8f2f1fd9fef25bf1f9d8b329e7d2bc1bbf8f7da339cff39"}
Jul 28 17:09:35.401319 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:35.401306    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh" event={"ID":"b0a6ecd5-1b96-4003-ba79-fe147a67c757","Type":"ContainerStarted","Data":"ca582e4073bdb39952a530643a8755add85099acb3c42ff982a00da243411313"}
Jul 28 17:09:35.403087 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:35.403031    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg" event={"ID":"c643884f-420e-452c-98b0-09aede5dd675","Type":"ContainerStarted","Data":"d7767276ad7c2e1ef4acaf44f99e415c8d4aa1d0598c6737f7b9d08bd859ff41"}
Jul 28 17:09:35.422574 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:35.422530    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pp6xv" podStartSLOduration=4.699599425 podStartE2EDuration="23.422513631s" totalImagesPullingTime="18.722914206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:35.422140576 +0000 UTC m=+23.696757009" watchObservedRunningTime="2026-07-28 17:09:35.422513631 +0000 UTC m=+23.697130066"
Jul 28 17:09:36.221308 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:36.221278    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:36.285004 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:36.284977    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:36.285110 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:36.285011    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:09:36.285110 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:36.284976    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:36.285201 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:36.285179    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkf8x" podUID="9a7b47e4-993e-4527-a00f-4ebbbc050348"
Jul 28 17:09:36.285313 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:36.285292    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nt29" podUID="33068506-3820-4d64-bc6b-33ea180dca70"
Jul 28 17:09:36.285404 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:36.285387    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwmpm" podUID="189dbf08-647d-42a8-975f-39838ad4edf9"
Jul 28 17:09:36.306358 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:36.306327    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:36.407321 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:36.407285    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg" event={"ID":"c643884f-420e-452c-98b0-09aede5dd675","Type":"ContainerStarted","Data":"e89f8069751563150cfad865319a0d81f3f55cca982afba03850bc82467659ed"}
Jul 28 17:09:36.427724 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:36.427675    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t82zg" podStartSLOduration=3.9578539470000003 podStartE2EDuration="24.427655953s" totalImagesPullingTime="20.469802006s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:36.42575727 +0000 UTC m=+24.700373703" watchObservedRunningTime="2026-07-28 17:09:36.427655953 +0000 UTC m=+24.702272387"
Jul 28 17:09:37.412874 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:37.412683    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 17:09:37.413336 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:37.413306    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh" event={"ID":"b0a6ecd5-1b96-4003-ba79-fe147a67c757","Type":"ContainerStarted","Data":"b54d953874c5c6905a1f4a532685a603935d2706d9cda6a94e018e10c26327e9"}
Jul 28 17:09:38.285167 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:38.285135    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:38.285167 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:38.285147    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:09:38.285432 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:38.285138    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:38.285432 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:38.285278    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkf8x" podUID="9a7b47e4-993e-4527-a00f-4ebbbc050348"
Jul 28 17:09:38.285432 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:38.285320    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwmpm" podUID="189dbf08-647d-42a8-975f-39838ad4edf9"
Jul 28 17:09:38.285432 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:38.285378    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nt29" podUID="33068506-3820-4d64-bc6b-33ea180dca70"
Jul 28 17:09:38.953724 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:38.953698    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kc9pw"
Jul 28 17:09:38.954470 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:38.954452    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kc9pw"
Jul 28 17:09:39.420720 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:39.420544    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 17:09:39.421063 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:39.421032    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh" event={"ID":"b0a6ecd5-1b96-4003-ba79-fe147a67c757","Type":"ContainerStarted","Data":"0b8ee4b2eb20b8e38eba139b8ff0c7c8ec68443ca6965c68cbe198b51c4deec6"}
Jul 28 17:09:39.421719 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:39.421321    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:39.421719 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:39.421383    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:39.421719 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:39.421430    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:39.421719 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:39.421456    2581 scope.go:117] "RemoveContainer" containerID="adc1996482b2b66affd4f664792f231db443e6be0c4a22eaaa4fc38ab1c7a7fe"
Jul 28 17:09:39.423126 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:39.423093    2581 generic.go:356] "Generic (PLEG): container finished" podID="1e459f08-3f93-48bd-bb45-b950f416c87f" containerID="b879ec7bce976bb4b9dff8afbfc7f00a224eb97aee4332437171e5bce367408b" exitCode=0
Jul 28 17:09:39.423221 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:39.423200    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njk8v" event={"ID":"1e459f08-3f93-48bd-bb45-b950f416c87f","Type":"ContainerDied","Data":"b879ec7bce976bb4b9dff8afbfc7f00a224eb97aee4332437171e5bce367408b"}
Jul 28 17:09:39.437449 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:39.437422    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:39.438716 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:39.438700    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:09:39.756936 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:39.756850    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kc9pw"
Jul 28 17:09:39.757283 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:39.757259    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kc9pw"
Jul 28 17:09:39.853305 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:39.853278    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:40.285572 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:40.285544    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:40.286066 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:40.285546    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:40.286066 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:40.285678    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:09:40.286066 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:40.285675    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwmpm" podUID="189dbf08-647d-42a8-975f-39838ad4edf9"
Jul 28 17:09:40.286066 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:40.285748    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkf8x" podUID="9a7b47e4-993e-4527-a00f-4ebbbc050348"
Jul 28 17:09:40.286066 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:40.285805    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nt29" podUID="33068506-3820-4d64-bc6b-33ea180dca70"
Jul 28 17:09:40.429933 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:40.429906    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 17:09:40.430371 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:40.430346    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh" event={"ID":"b0a6ecd5-1b96-4003-ba79-fe147a67c757","Type":"ContainerStarted","Data":"71759d29088f8cc75a1686930ab166990feb0c4561f19b44689e06c48c47b4f6"}
Jul 28 17:09:40.464914 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:40.464853    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh" podStartSLOduration=9.69355004 podStartE2EDuration="28.464840532s" totalImagesPullingTime="18.771290492s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:40.464652295 +0000 UTC m=+28.739268727" watchObservedRunningTime="2026-07-28 17:09:40.464840532 +0000 UTC m=+28.739456964"
Jul 28 17:09:40.694384 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:40.694350    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:40.769851 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:40.769824    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:40.870581 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:40.870535    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:41.208821 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:41.208622    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jwmpm"]
Jul 28 17:09:41.209022 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:41.208855    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:41.209022 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:41.208957    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwmpm" podUID="189dbf08-647d-42a8-975f-39838ad4edf9"
Jul 28 17:09:41.211359 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:41.211333    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2nt29"]
Jul 28 17:09:41.211487 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:41.211429    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:09:41.211547 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:41.211517    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nt29" podUID="33068506-3820-4d64-bc6b-33ea180dca70"
Jul 28 17:09:41.212181 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:41.212147    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zkf8x"]
Jul 28 17:09:41.212281 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:41.212241    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:41.212394 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:41.212358    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkf8x" podUID="9a7b47e4-993e-4527-a00f-4ebbbc050348"
Jul 28 17:09:41.434053 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:41.434024    2581 generic.go:356] "Generic (PLEG): container finished" podID="1e459f08-3f93-48bd-bb45-b950f416c87f" containerID="1a8f55fbdf9c21f7bc31ec97d4c80617f8687e3452a8a013024127ba4b87d38b" exitCode=0
Jul 28 17:09:41.434498 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:41.434112    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njk8v" event={"ID":"1e459f08-3f93-48bd-bb45-b950f416c87f","Type":"ContainerDied","Data":"1a8f55fbdf9c21f7bc31ec97d4c80617f8687e3452a8a013024127ba4b87d38b"}
Jul 28 17:09:41.757271 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:41.757230    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:41.833789 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:41.833759    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:42.272128 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:42.272097    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:42.372716 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:42.372689    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:43.285054 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:43.285031    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:43.285596 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:43.285125    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:09:43.285596 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:43.285132    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkf8x" podUID="9a7b47e4-993e-4527-a00f-4ebbbc050348"
Jul 28 17:09:43.285596 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:43.285156    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:43.285596 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:43.285224    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nt29" podUID="33068506-3820-4d64-bc6b-33ea180dca70"
Jul 28 17:09:43.285596 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:43.285285    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwmpm" podUID="189dbf08-647d-42a8-975f-39838ad4edf9"
Jul 28 17:09:43.441231 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:43.441197    2581 generic.go:356] "Generic (PLEG): container finished" podID="1e459f08-3f93-48bd-bb45-b950f416c87f" containerID="627fa4bc7a3e3a2313b424161e82078463f32d5d97a4e754b3d208f8e8087901" exitCode=0
Jul 28 17:09:43.441354 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:43.441271    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njk8v" event={"ID":"1e459f08-3f93-48bd-bb45-b950f416c87f","Type":"ContainerDied","Data":"627fa4bc7a3e3a2313b424161e82078463f32d5d97a4e754b3d208f8e8087901"}
Jul 28 17:09:43.727399 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:43.727368    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:45.039333 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:45.039296    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:45.149105 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:45.149074    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:45.285311 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:45.285281    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:09:45.285455 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:45.285393    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:45.285455 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:45.285407    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2nt29" podUID="33068506-3820-4d64-bc6b-33ea180dca70"
Jul 28 17:09:45.285455 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:45.285439    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:45.285573 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:45.285547    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zkf8x" podUID="9a7b47e4-993e-4527-a00f-4ebbbc050348"
Jul 28 17:09:45.285651 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:45.285634    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwmpm" podUID="189dbf08-647d-42a8-975f-39838ad4edf9"
Jul 28 17:09:45.376585 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:45.376510    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:45.476402 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:45.476374    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:45.577162 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:45.577128    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:45.898240 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:45.898211    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgzn7\" (UniqueName: \"kubernetes.io/projected/33068506-3820-4d64-bc6b-33ea180dca70-kube-api-access-jgzn7\") pod \"network-check-target-2nt29\" (UID: \"33068506-3820-4d64-bc6b-33ea180dca70\") " pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:09:45.898391 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:45.898248    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a7b47e4-993e-4527-a00f-4ebbbc050348-metrics-certs\") pod \"network-metrics-daemon-zkf8x\" (UID: \"9a7b47e4-993e-4527-a00f-4ebbbc050348\") " pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:45.898391 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:45.898269    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/189dbf08-647d-42a8-975f-39838ad4edf9-original-pull-secret\") pod \"global-pull-secret-syncer-jwmpm\" (UID: \"189dbf08-647d-42a8-975f-39838ad4edf9\") " pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:45.898391 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:45.898351    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:45.898391 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:45.898358    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:45.898391 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:45.898371    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:09:45.898391 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:45.898388    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:09:45.898567 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:45.898397    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/189dbf08-647d-42a8-975f-39838ad4edf9-original-pull-secret podName:189dbf08-647d-42a8-975f-39838ad4edf9 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:17.89838411 +0000 UTC m=+66.173000520 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/189dbf08-647d-42a8-975f-39838ad4edf9-original-pull-secret") pod "global-pull-secret-syncer-jwmpm" (UID: "189dbf08-647d-42a8-975f-39838ad4edf9") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:09:45.898567 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:45.898401    2581 projected.go:196] Error preparing data for projected volume kube-api-access-jgzn7 for pod openshift-network-diagnostics/network-check-target-2nt29: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:45.898567 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:45.898418    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a7b47e4-993e-4527-a00f-4ebbbc050348-metrics-certs podName:9a7b47e4-993e-4527-a00f-4ebbbc050348 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:17.898401353 +0000 UTC m=+66.173017776 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a7b47e4-993e-4527-a00f-4ebbbc050348-metrics-certs") pod "network-metrics-daemon-zkf8x" (UID: "9a7b47e4-993e-4527-a00f-4ebbbc050348") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:09:45.898567 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:45.898437    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33068506-3820-4d64-bc6b-33ea180dca70-kube-api-access-jgzn7 podName:33068506-3820-4d64-bc6b-33ea180dca70 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:17.898426548 +0000 UTC m=+66.173042977 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgzn7" (UniqueName: "kubernetes.io/projected/33068506-3820-4d64-bc6b-33ea180dca70-kube-api-access-jgzn7") pod "network-check-target-2nt29" (UID: "33068506-3820-4d64-bc6b-33ea180dca70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:09:46.324369 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.324335    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:46.425551 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.425513    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:46.489844 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.489813    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-231.ec2.internal" event="NodeReady"
Jul 28 17:09:46.490013 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.489973    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 17:09:46.536527 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.536499    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mvdj9"]
Jul 28 17:09:46.541729 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.541705    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-h6pjb"]
Jul 28 17:09:46.541862 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.541835    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mvdj9"
Jul 28 17:09:46.544591 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.544561    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 17:09:46.544702 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.544622    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xtzb7\""
Jul 28 17:09:46.544930 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.544909    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 17:09:46.545440 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.545418    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h6pjb"
Jul 28 17:09:46.547884 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.547862    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h6cgw\""
Jul 28 17:09:46.548426 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.548302    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 17:09:46.548530 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.548424    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 17:09:46.548936 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.548781    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 17:09:46.550153 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.550134    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mvdj9"]
Jul 28 17:09:46.565429 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.565407    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h6pjb"]
Jul 28 17:09:46.690714 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.690685    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:46.704571 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.704546    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk4q8\" (UniqueName: \"kubernetes.io/projected/81dc569e-3e52-4bb3-8f79-d58b183bb100-kube-api-access-mk4q8\") pod \"dns-default-mvdj9\" (UID: \"81dc569e-3e52-4bb3-8f79-d58b183bb100\") " pod="openshift-dns/dns-default-mvdj9"
Jul 28 17:09:46.704691 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.704599    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/baf5cf61-87a1-442f-981a-8d24b1197ec8-cert\") pod \"ingress-canary-h6pjb\" (UID: \"baf5cf61-87a1-442f-981a-8d24b1197ec8\") " pod="openshift-ingress-canary/ingress-canary-h6pjb"
Jul 28 17:09:46.704691 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.704656    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81dc569e-3e52-4bb3-8f79-d58b183bb100-tmp-dir\") pod \"dns-default-mvdj9\" (UID: \"81dc569e-3e52-4bb3-8f79-d58b183bb100\") " pod="openshift-dns/dns-default-mvdj9"
Jul 28 17:09:46.704691 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.704681    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81dc569e-3e52-4bb3-8f79-d58b183bb100-metrics-tls\") pod \"dns-default-mvdj9\" (UID: \"81dc569e-3e52-4bb3-8f79-d58b183bb100\") " pod="openshift-dns/dns-default-mvdj9"
Jul 28 17:09:46.704841 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.704699    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/81dc569e-3e52-4bb3-8f79-d58b183bb100-config-volume\") pod \"dns-default-mvdj9\" (UID: \"81dc569e-3e52-4bb3-8f79-d58b183bb100\") " pod="openshift-dns/dns-default-mvdj9"
Jul 28 17:09:46.704841 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.704725    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvcvp\" (UniqueName: \"kubernetes.io/projected/baf5cf61-87a1-442f-981a-8d24b1197ec8-kube-api-access-hvcvp\") pod \"ingress-canary-h6pjb\" (UID: \"baf5cf61-87a1-442f-981a-8d24b1197ec8\") " pod="openshift-ingress-canary/ingress-canary-h6pjb"
Jul 28 17:09:46.791907 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.791862    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:46.805171 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.805130    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvcvp\" (UniqueName: \"kubernetes.io/projected/baf5cf61-87a1-442f-981a-8d24b1197ec8-kube-api-access-hvcvp\") pod \"ingress-canary-h6pjb\" (UID: \"baf5cf61-87a1-442f-981a-8d24b1197ec8\") " pod="openshift-ingress-canary/ingress-canary-h6pjb"
Jul 28 17:09:46.805317 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.805209    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk4q8\" (UniqueName: \"kubernetes.io/projected/81dc569e-3e52-4bb3-8f79-d58b183bb100-kube-api-access-mk4q8\") pod \"dns-default-mvdj9\" (UID: \"81dc569e-3e52-4bb3-8f79-d58b183bb100\") " pod="openshift-dns/dns-default-mvdj9"
Jul 28 17:09:46.805317 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.805256    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/baf5cf61-87a1-442f-981a-8d24b1197ec8-cert\") pod \"ingress-canary-h6pjb\" (UID: \"baf5cf61-87a1-442f-981a-8d24b1197ec8\") " pod="openshift-ingress-canary/ingress-canary-h6pjb"
Jul 28 17:09:46.805317 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.805305    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81dc569e-3e52-4bb3-8f79-d58b183bb100-tmp-dir\") pod \"dns-default-mvdj9\" (UID: \"81dc569e-3e52-4bb3-8f79-d58b183bb100\") " pod="openshift-dns/dns-default-mvdj9"
Jul 28 17:09:46.805476 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.805329    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81dc569e-3e52-4bb3-8f79-d58b183bb100-metrics-tls\") pod \"dns-default-mvdj9\" (UID: \"81dc569e-3e52-4bb3-8f79-d58b183bb100\") " pod="openshift-dns/dns-default-mvdj9"
Jul 28 17:09:46.805476 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.805347    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/81dc569e-3e52-4bb3-8f79-d58b183bb100-config-volume\") pod \"dns-default-mvdj9\" (UID: \"81dc569e-3e52-4bb3-8f79-d58b183bb100\") " pod="openshift-dns/dns-default-mvdj9"
Jul 28 17:09:46.805585 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:46.805487    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:09:46.805585 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:46.805538    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dc569e-3e52-4bb3-8f79-d58b183bb100-metrics-tls podName:81dc569e-3e52-4bb3-8f79-d58b183bb100 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:47.305523272 +0000 UTC m=+35.580139683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81dc569e-3e52-4bb3-8f79-d58b183bb100-metrics-tls") pod "dns-default-mvdj9" (UID: "81dc569e-3e52-4bb3-8f79-d58b183bb100") : secret "dns-default-metrics-tls" not found
Jul 28 17:09:46.805717 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.805696    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81dc569e-3e52-4bb3-8f79-d58b183bb100-tmp-dir\") pod \"dns-default-mvdj9\" (UID: \"81dc569e-3e52-4bb3-8f79-d58b183bb100\") " pod="openshift-dns/dns-default-mvdj9"
Jul 28 17:09:46.805950 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.805910    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/81dc569e-3e52-4bb3-8f79-d58b183bb100-config-volume\") pod \"dns-default-mvdj9\" (UID: \"81dc569e-3e52-4bb3-8f79-d58b183bb100\") " pod="openshift-dns/dns-default-mvdj9"
Jul 28 17:09:46.809981 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.809963    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/baf5cf61-87a1-442f-981a-8d24b1197ec8-cert\") pod \"ingress-canary-h6pjb\" (UID: \"baf5cf61-87a1-442f-981a-8d24b1197ec8\") " pod="openshift-ingress-canary/ingress-canary-h6pjb"
Jul 28 17:09:46.814972 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.814932    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk4q8\" (UniqueName: \"kubernetes.io/projected/81dc569e-3e52-4bb3-8f79-d58b183bb100-kube-api-access-mk4q8\") pod \"dns-default-mvdj9\" (UID: \"81dc569e-3e52-4bb3-8f79-d58b183bb100\") " pod="openshift-dns/dns-default-mvdj9"
Jul 28 17:09:46.815393 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.815371    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvcvp\" (UniqueName: \"kubernetes.io/projected/baf5cf61-87a1-442f-981a-8d24b1197ec8-kube-api-access-hvcvp\") pod \"ingress-canary-h6pjb\" (UID: \"baf5cf61-87a1-442f-981a-8d24b1197ec8\") " pod="openshift-ingress-canary/ingress-canary-h6pjb"
Jul 28 17:09:46.859988 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:46.859963    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-h6pjb"
Jul 28 17:09:47.050118 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:47.050031    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:47.154836 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:47.154801    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:47.285125 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:47.285096    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:09:47.285295 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:47.285215    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:09:47.285295 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:47.285254    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:09:47.288688 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:47.288663    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:09:47.288815 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:47.288745    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:09:47.288815 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:47.288759    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:09:47.288815 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:47.288790    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:09:47.290151 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:47.290129    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b4dt6\""
Jul 28 17:09:47.290304 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:47.290239    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7zr5q\""
Jul 28 17:09:47.308253 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:47.308219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81dc569e-3e52-4bb3-8f79-d58b183bb100-metrics-tls\") pod \"dns-default-mvdj9\" (UID: \"81dc569e-3e52-4bb3-8f79-d58b183bb100\") " pod="openshift-dns/dns-default-mvdj9"
Jul 28 17:09:47.308423 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:47.308387    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:09:47.308513 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:47.308456    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dc569e-3e52-4bb3-8f79-d58b183bb100-metrics-tls podName:81dc569e-3e52-4bb3-8f79-d58b183bb100 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:48.308438647 +0000 UTC m=+36.583055057 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81dc569e-3e52-4bb3-8f79-d58b183bb100-metrics-tls") pod "dns-default-mvdj9" (UID: "81dc569e-3e52-4bb3-8f79-d58b183bb100") : secret "dns-default-metrics-tls" not found
Jul 28 17:09:47.542886 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:47.542858    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-h6pjb"]
Jul 28 17:09:47.546256 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:47.546233    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf5cf61_87a1_442f_981a_8d24b1197ec8.slice/crio-9f3ff7f37c40888d698e3ef0e31dca817e595073b263301a176822a75d690b62 WatchSource:0}: Error finding container 9f3ff7f37c40888d698e3ef0e31dca817e595073b263301a176822a75d690b62: Status 404 returned error can't find the container with id 9f3ff7f37c40888d698e3ef0e31dca817e595073b263301a176822a75d690b62
Jul 28 17:09:47.983802 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:47.983770    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:48.317387 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:48.317302    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81dc569e-3e52-4bb3-8f79-d58b183bb100-metrics-tls\") pod \"dns-default-mvdj9\" (UID: \"81dc569e-3e52-4bb3-8f79-d58b183bb100\") " pod="openshift-dns/dns-default-mvdj9"
Jul 28 17:09:48.317515 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:48.317439    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:09:48.317554 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:48.317515    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dc569e-3e52-4bb3-8f79-d58b183bb100-metrics-tls podName:81dc569e-3e52-4bb3-8f79-d58b183bb100 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:50.317495685 +0000 UTC m=+38.592112100 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81dc569e-3e52-4bb3-8f79-d58b183bb100-metrics-tls") pod "dns-default-mvdj9" (UID: "81dc569e-3e52-4bb3-8f79-d58b183bb100") : secret "dns-default-metrics-tls" not found
Jul 28 17:09:48.457916 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:48.457879    2581 generic.go:356] "Generic (PLEG): container finished" podID="1e459f08-3f93-48bd-bb45-b950f416c87f" containerID="1cb0850618d08086f0d52c4a695047381c16493cbd3237895f17232776638a11" exitCode=0
Jul 28 17:09:48.458113 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:48.457977    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njk8v" event={"ID":"1e459f08-3f93-48bd-bb45-b950f416c87f","Type":"ContainerDied","Data":"1cb0850618d08086f0d52c4a695047381c16493cbd3237895f17232776638a11"}
Jul 28 17:09:48.459305 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:48.459287    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h6pjb" event={"ID":"baf5cf61-87a1-442f-981a-8d24b1197ec8","Type":"ContainerStarted","Data":"9f3ff7f37c40888d698e3ef0e31dca817e595073b263301a176822a75d690b62"}
Jul 28 17:09:48.623439 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:48.623409    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-blvc4"]
Jul 28 17:09:48.626740 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:48.626720    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-blvc4"
Jul 28 17:09:48.629577 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:48.629552    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 28 17:09:48.629577 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:48.629563    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 28 17:09:48.629726 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:48.629691    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 28 17:09:48.629902 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:48.629886    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 28 17:09:48.630125 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:48.630111    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-vjnl7\""
Jul 28 17:09:48.634672 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:48.634560    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-blvc4"]
Jul 28 17:09:48.668469 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:48.668444    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:48.720779 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:48.720743    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfpvz\" (UniqueName: \"kubernetes.io/projected/72963718-7324-4d58-92a9-d026d6938c96-kube-api-access-wfpvz\") pod \"service-ca-658d7b7c78-blvc4\" (UID: \"72963718-7324-4d58-92a9-d026d6938c96\") " pod="openshift-service-ca/service-ca-658d7b7c78-blvc4"
Jul 28 17:09:48.720779 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:48.720784    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/72963718-7324-4d58-92a9-d026d6938c96-signing-cabundle\") pod \"service-ca-658d7b7c78-blvc4\" (UID: \"72963718-7324-4d58-92a9-d026d6938c96\") " pod="openshift-service-ca/service-ca-658d7b7c78-blvc4"
Jul 28 17:09:48.721015 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:48.720846    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/72963718-7324-4d58-92a9-d026d6938c96-signing-key\") pod \"service-ca-658d7b7c78-blvc4\" (UID: \"72963718-7324-4d58-92a9-d026d6938c96\") " pod="openshift-service-ca/service-ca-658d7b7c78-blvc4"
Jul 28 17:09:48.732534 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:48.732504    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:48.822240 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:48.822155    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfpvz\" (UniqueName: \"kubernetes.io/projected/72963718-7324-4d58-92a9-d026d6938c96-kube-api-access-wfpvz\") pod \"service-ca-658d7b7c78-blvc4\" (UID: \"72963718-7324-4d58-92a9-d026d6938c96\") " pod="openshift-service-ca/service-ca-658d7b7c78-blvc4"
Jul 28 17:09:48.822240 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:48.822201    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/72963718-7324-4d58-92a9-d026d6938c96-signing-cabundle\") pod \"service-ca-658d7b7c78-blvc4\" (UID: \"72963718-7324-4d58-92a9-d026d6938c96\") " pod="openshift-service-ca/service-ca-658d7b7c78-blvc4"
Jul 28 17:09:48.822453 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:48.822290    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/72963718-7324-4d58-92a9-d026d6938c96-signing-key\") pod \"service-ca-658d7b7c78-blvc4\" (UID: \"72963718-7324-4d58-92a9-d026d6938c96\") " pod="openshift-service-ca/service-ca-658d7b7c78-blvc4"
Jul 28 17:09:48.823019 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:48.822996    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/72963718-7324-4d58-92a9-d026d6938c96-signing-cabundle\") pod \"service-ca-658d7b7c78-blvc4\" (UID: \"72963718-7324-4d58-92a9-d026d6938c96\") " pod="openshift-service-ca/service-ca-658d7b7c78-blvc4"
Jul 28 17:09:48.826541 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:48.826518    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/72963718-7324-4d58-92a9-d026d6938c96-signing-key\") pod \"service-ca-658d7b7c78-blvc4\" (UID: \"72963718-7324-4d58-92a9-d026d6938c96\") " pod="openshift-service-ca/service-ca-658d7b7c78-blvc4"
Jul 28 17:09:48.831511 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:48.831484    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfpvz\" (UniqueName: \"kubernetes.io/projected/72963718-7324-4d58-92a9-d026d6938c96-kube-api-access-wfpvz\") pod \"service-ca-658d7b7c78-blvc4\" (UID: \"72963718-7324-4d58-92a9-d026d6938c96\") " pod="openshift-service-ca/service-ca-658d7b7c78-blvc4"
Jul 28 17:09:48.938013 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:48.937977    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-blvc4"
Jul 28 17:09:49.465044 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:49.464761    2581 generic.go:356] "Generic (PLEG): container finished" podID="1e459f08-3f93-48bd-bb45-b950f416c87f" containerID="3a25ea8a93c4b2258f78721f68d63576c37743922ad88cd73c70d26679be472b" exitCode=0
Jul 28 17:09:49.465044 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:49.464848    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njk8v" event={"ID":"1e459f08-3f93-48bd-bb45-b950f416c87f","Type":"ContainerDied","Data":"3a25ea8a93c4b2258f78721f68d63576c37743922ad88cd73c70d26679be472b"}
Jul 28 17:09:49.595306 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:49.595271    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:49.783478 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:49.783446    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-blvc4"]
Jul 28 17:09:49.786154 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:09:49.786129    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72963718_7324_4d58_92a9_d026d6938c96.slice/crio-ae6476b3d131348a8ea8b61b68acab6bc227ebfc3d2ba326b1b7bc6e841a0a49 WatchSource:0}: Error finding container ae6476b3d131348a8ea8b61b68acab6bc227ebfc3d2ba326b1b7bc6e841a0a49: Status 404 returned error can't find the container with id ae6476b3d131348a8ea8b61b68acab6bc227ebfc3d2ba326b1b7bc6e841a0a49
Jul 28 17:09:50.335044 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:50.334968    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81dc569e-3e52-4bb3-8f79-d58b183bb100-metrics-tls\") pod \"dns-default-mvdj9\" (UID: \"81dc569e-3e52-4bb3-8f79-d58b183bb100\") " pod="openshift-dns/dns-default-mvdj9"
Jul 28 17:09:50.335187 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:50.335115    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:09:50.335187 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:50.335182    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dc569e-3e52-4bb3-8f79-d58b183bb100-metrics-tls podName:81dc569e-3e52-4bb3-8f79-d58b183bb100 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:54.335155866 +0000 UTC m=+42.609772294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81dc569e-3e52-4bb3-8f79-d58b183bb100-metrics-tls") pod "dns-default-mvdj9" (UID: "81dc569e-3e52-4bb3-8f79-d58b183bb100") : secret "dns-default-metrics-tls" not found
Jul 28 17:09:50.469484 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:50.469445    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-blvc4" event={"ID":"72963718-7324-4d58-92a9-d026d6938c96","Type":"ContainerStarted","Data":"ae6476b3d131348a8ea8b61b68acab6bc227ebfc3d2ba326b1b7bc6e841a0a49"}
Jul 28 17:09:50.473426 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:50.473398    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-njk8v" event={"ID":"1e459f08-3f93-48bd-bb45-b950f416c87f","Type":"ContainerStarted","Data":"b52b7abfa024d5209f3ef6572e38b753443233929a094c456dc0f78dd4583257"}
Jul 28 17:09:50.474868 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:50.474840    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-h6pjb" event={"ID":"baf5cf61-87a1-442f-981a-8d24b1197ec8","Type":"ContainerStarted","Data":"71706f3d591e59ebdd0afa7ed77cf6bba3f6b9b97bd72bd3627ad3759b099283"}
Jul 28 17:09:50.503604 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:50.503561    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-njk8v" podStartSLOduration=8.768431203 podStartE2EDuration="38.503544355s" totalImagesPullingTime="29.735113152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:50.503452687 +0000 UTC m=+38.778069121" watchObservedRunningTime="2026-07-28 17:09:50.503544355 +0000 UTC m=+38.778160789"
Jul 28 17:09:50.521780 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:50.521745    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-h6pjb" podStartSLOduration=2.108776025 podStartE2EDuration="4.521731717s" totalImagesPullingTime="2.412955692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:50.52067444 +0000 UTC m=+38.795290875" watchObservedRunningTime="2026-07-28 17:09:50.521731717 +0000 UTC m=+38.796348148"
Jul 28 17:09:52.258931 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:52.258887    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:52.359871 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:52.359841    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:52.460734 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:52.460631    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:52.481101 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:52.481064    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-blvc4" event={"ID":"72963718-7324-4d58-92a9-d026d6938c96","Type":"ContainerStarted","Data":"a9f3ed0d83182cad8e7cabd354b1c4544c40ab34155ca43870cb6d8c32e1a073"}
Jul 28 17:09:52.503180 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:52.503128    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-blvc4" podStartSLOduration=2.119442756 podStartE2EDuration="4.503109858s" totalImagesPullingTime="2.383667102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:52.501558931 +0000 UTC m=+40.776175368" watchObservedRunningTime="2026-07-28 17:09:52.503109858 +0000 UTC m=+40.777726291"
Jul 28 17:09:54.044364 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:54.044326    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:54.128415 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:54.128382    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:54.227310 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:54.227274    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:54.331123 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:54.331025    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:54.365110 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:54.364388    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81dc569e-3e52-4bb3-8f79-d58b183bb100-metrics-tls\") pod \"dns-default-mvdj9\" (UID: \"81dc569e-3e52-4bb3-8f79-d58b183bb100\") " pod="openshift-dns/dns-default-mvdj9"
Jul 28 17:09:54.365110 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:54.364518    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:09:54.365110 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:09:54.364583    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dc569e-3e52-4bb3-8f79-d58b183bb100-metrics-tls podName:81dc569e-3e52-4bb3-8f79-d58b183bb100 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:02.364565889 +0000 UTC m=+50.639182314 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81dc569e-3e52-4bb3-8f79-d58b183bb100-metrics-tls") pod "dns-default-mvdj9" (UID: "81dc569e-3e52-4bb3-8f79-d58b183bb100") : secret "dns-default-metrics-tls" not found
Jul 28 17:09:54.654609 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:54.654571    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:55.641077 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:55.641044    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:55.844125 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:55.844077    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:56.050708 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:56.050673    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:56.454427 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:56.454387    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:57.041123 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:57.041082    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:57.358144 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:57.358063    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:57.459080 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:57.459050    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:57.535516 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:57.535487    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:57.641306 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:57.641236    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:57.846090 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:57.846057    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:57.913739 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:57.913708    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:58.237023 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:58.236924    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:58.445080 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:58.445043    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:58.510543 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:58.510463    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:58.843160 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:58.843085    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:59.439720 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:59.439688    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:59.641019 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:59.640986    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:09:59.802604 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:09:59.802532    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:00.072823 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:00.072743    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:00.199242 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:00.199191    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:00.372627 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:00.372556    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:00.495324 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:00.495292    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:00.599488 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:00.599460    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:00.699226 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:00.699199    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:00.876266 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:00.876235    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:00.936846 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:00.936822    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:01.088006 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:01.087931    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:01.188390 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:01.188363    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:01.289681 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:01.289645    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:01.358278 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:01.358205    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:01.647140 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:01.647060    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:01.705318 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:01.705290    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:02.080049 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:02.080015    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:02.155933 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:02.155903    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:02.251452 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:02.251423    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:02.420887 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:02.420857    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81dc569e-3e52-4bb3-8f79-d58b183bb100-metrics-tls\") pod \"dns-default-mvdj9\" (UID: \"81dc569e-3e52-4bb3-8f79-d58b183bb100\") " pod="openshift-dns/dns-default-mvdj9"
Jul 28 17:10:02.424152 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:02.424131    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81dc569e-3e52-4bb3-8f79-d58b183bb100-metrics-tls\") pod \"dns-default-mvdj9\" (UID: \"81dc569e-3e52-4bb3-8f79-d58b183bb100\") " pod="openshift-dns/dns-default-mvdj9"
Jul 28 17:10:02.455116 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:02.455091    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mvdj9"
Jul 28 17:10:02.653144 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:02.653101    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mvdj9"]
Jul 28 17:10:02.656964 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:10:02.656923    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81dc569e_3e52_4bb3_8f79_d58b183bb100.slice/crio-de263e4b7042a9dc573bd7a860ee4f27762227d3a3971897d7c20051335ea974 WatchSource:0}: Error finding container de263e4b7042a9dc573bd7a860ee4f27762227d3a3971897d7c20051335ea974: Status 404 returned error can't find the container with id de263e4b7042a9dc573bd7a860ee4f27762227d3a3971897d7c20051335ea974
Jul 28 17:10:03.099589 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:03.099554    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:03.162103 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:03.162078    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:03.446225 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:03.446198    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:03.507356 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:03.507324    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mvdj9" event={"ID":"81dc569e-3e52-4bb3-8f79-d58b183bb100","Type":"ContainerStarted","Data":"de263e4b7042a9dc573bd7a860ee4f27762227d3a3971897d7c20051335ea974"}
Jul 28 17:10:03.638350 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:03.638315    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:03.844307 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:03.844233    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:04.046363 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:04.046329    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:04.288418 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:04.288387    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:04.657337 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:04.657305    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:05.513310 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:05.513287    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mvdj9" event={"ID":"81dc569e-3e52-4bb3-8f79-d58b183bb100","Type":"ContainerStarted","Data":"54684b87347cc910ca1b61e30218f171d2118a546ebad8484ebd5a44cf28bbec"}
Jul 28 17:10:06.517798 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:06.517764    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mvdj9" event={"ID":"81dc569e-3e52-4bb3-8f79-d58b183bb100","Type":"ContainerStarted","Data":"7138a04ed0cd0a604ce3ef4e7ba56f8b54bbcf3afda5c2c161b94be4646b9e2b"}
Jul 28 17:10:06.518281 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:06.517892    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mvdj9"
Jul 28 17:10:06.536959 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:06.536904    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mvdj9" podStartSLOduration=17.888918499 podStartE2EDuration="20.536891623s" totalImagesPullingTime="2.647973124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:06.536144865 +0000 UTC m=+54.810761297" watchObservedRunningTime="2026-07-28 17:10:06.536891623 +0000 UTC m=+54.811508055"
Jul 28 17:10:09.156472 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.156434    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8896d549-vgrc4"]
Jul 28 17:10:09.159567 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.159544    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8896d549-vgrc4"
Jul 28 17:10:09.160597 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.160576    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9b97dbc5-v2nc6"]
Jul 28 17:10:09.162390 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.162370    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 28 17:10:09.162476 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.162373    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 17:10:09.163453 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.163431    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9b97dbc5-v2nc6"
Jul 28 17:10:09.163619 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.163601    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 17:10:09.163682 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.163633    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 17:10:09.165901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.165871    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 28 17:10:09.166103 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.166074    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 28 17:10:09.166207 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.166187    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 28 17:10:09.166272 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.166190    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 28 17:10:09.172597 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.172577    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8896d549-vgrc4"]
Jul 28 17:10:09.175973 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.175101    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9b97dbc5-v2nc6"]
Jul 28 17:10:09.254455 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.254429    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-fb6888f7-cwxc9"]
Jul 28 17:10:09.257416 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.257399    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5555ff8d6c-6mzsx"]
Jul 28 17:10:09.257550 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.257535    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb6888f7-cwxc9"
Jul 28 17:10:09.260357 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.260335    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5555ff8d6c-6mzsx"
Jul 28 17:10:09.266402 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.266383    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4lj9\" (UniqueName: \"kubernetes.io/projected/29817449-b2c8-40eb-9db7-c3d0e051c4ed-kube-api-access-q4lj9\") pod \"klusterlet-addon-workmgr-5d8896d549-vgrc4\" (UID: \"29817449-b2c8-40eb-9db7-c3d0e051c4ed\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8896d549-vgrc4"
Jul 28 17:10:09.266520 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.266412    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29817449-b2c8-40eb-9db7-c3d0e051c4ed-tmp\") pod \"klusterlet-addon-workmgr-5d8896d549-vgrc4\" (UID: \"29817449-b2c8-40eb-9db7-c3d0e051c4ed\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8896d549-vgrc4"
Jul 28 17:10:09.266520 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.266426    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6f9f5b85-a06a-489f-a8dd-3408da0a5e8a-ca\") pod \"cluster-proxy-proxy-agent-d9b97dbc5-v2nc6\" (UID: \"6f9f5b85-a06a-489f-a8dd-3408da0a5e8a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9b97dbc5-v2nc6"
Jul 28 17:10:09.266520 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.266497    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/29817449-b2c8-40eb-9db7-c3d0e051c4ed-klusterlet-config\") pod \"klusterlet-addon-workmgr-5d8896d549-vgrc4\" (UID: \"29817449-b2c8-40eb-9db7-c3d0e051c4ed\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8896d549-vgrc4"
Jul 28 17:10:09.266692 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.266538    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p644g\" (UniqueName: \"kubernetes.io/projected/6f9f5b85-a06a-489f-a8dd-3408da0a5e8a-kube-api-access-p644g\") pod \"cluster-proxy-proxy-agent-d9b97dbc5-v2nc6\" (UID: \"6f9f5b85-a06a-489f-a8dd-3408da0a5e8a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9b97dbc5-v2nc6"
Jul 28 17:10:09.266692 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.266573    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6f9f5b85-a06a-489f-a8dd-3408da0a5e8a-hub\") pod \"cluster-proxy-proxy-agent-d9b97dbc5-v2nc6\" (UID: \"6f9f5b85-a06a-489f-a8dd-3408da0a5e8a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9b97dbc5-v2nc6"
Jul 28 17:10:09.266692 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.266596    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6f9f5b85-a06a-489f-a8dd-3408da0a5e8a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d9b97dbc5-v2nc6\" (UID: \"6f9f5b85-a06a-489f-a8dd-3408da0a5e8a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9b97dbc5-v2nc6"
Jul 28 17:10:09.266692 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.266660    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f9f5b85-a06a-489f-a8dd-3408da0a5e8a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d9b97dbc5-v2nc6\" (UID: \"6f9f5b85-a06a-489f-a8dd-3408da0a5e8a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9b97dbc5-v2nc6"
Jul 28 17:10:09.266692 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.266684    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f9f5b85-a06a-489f-a8dd-3408da0a5e8a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d9b97dbc5-v2nc6\" (UID: \"6f9f5b85-a06a-489f-a8dd-3408da0a5e8a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9b97dbc5-v2nc6"
Jul 28 17:10:09.270380 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.270361    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 28 17:10:09.270473 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.270363    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 17:10:09.270535 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.270526    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 28 17:10:09.279642 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.279626    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 17:10:09.280113 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.280099    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 17:10:09.281200 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.281183    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 28 17:10:09.282562 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.282547    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 17:10:09.283182 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.283168    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hm5wg\""
Jul 28 17:10:09.283400 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.283388    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 17:10:09.298383 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.298360    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 28 17:10:09.298459 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.298416    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-gr8xx\""
Jul 28 17:10:09.301240 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.301218    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5555ff8d6c-6mzsx"]
Jul 28 17:10:09.322134 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.322114    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fb6888f7-cwxc9"]
Jul 28 17:10:09.328250 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.328230    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:09.338557 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.338534    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 17:10:09.356798 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.356775    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fb6888f7-cwxc9"]
Jul 28 17:10:09.356976 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:10:09.356933    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-qh45t registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-fb6888f7-cwxc9" podUID="1891b513-8781-471c-a563-00e9fa143d76"
Jul 28 17:10:09.367150 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.367129    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/29817449-b2c8-40eb-9db7-c3d0e051c4ed-klusterlet-config\") pod \"klusterlet-addon-workmgr-5d8896d549-vgrc4\" (UID: \"29817449-b2c8-40eb-9db7-c3d0e051c4ed\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8896d549-vgrc4"
Jul 28 17:10:09.367227 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.367162    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/496b869a-13ac-4171-8068-7c839c75b1e9-default-certificate\") pod \"router-default-5555ff8d6c-6mzsx\" (UID: \"496b869a-13ac-4171-8068-7c839c75b1e9\") " pod="openshift-ingress/router-default-5555ff8d6c-6mzsx"
Jul 28 17:10:09.367227 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.367186    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f9f5b85-a06a-489f-a8dd-3408da0a5e8a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d9b97dbc5-v2nc6\" (UID: \"6f9f5b85-a06a-489f-a8dd-3408da0a5e8a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9b97dbc5-v2nc6"
Jul 28 17:10:09.367227 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.367214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f9f5b85-a06a-489f-a8dd-3408da0a5e8a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d9b97dbc5-v2nc6\" (UID: \"6f9f5b85-a06a-489f-a8dd-3408da0a5e8a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9b97dbc5-v2nc6"
Jul 28 17:10:09.367367 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.367267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzsgr\" (UniqueName: \"kubernetes.io/projected/496b869a-13ac-4171-8068-7c839c75b1e9-kube-api-access-lzsgr\") pod \"router-default-5555ff8d6c-6mzsx\" (UID: \"496b869a-13ac-4171-8068-7c839c75b1e9\") " pod="openshift-ingress/router-default-5555ff8d6c-6mzsx"
Jul 28 17:10:09.367367 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.367297    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4lj9\" (UniqueName: \"kubernetes.io/projected/29817449-b2c8-40eb-9db7-c3d0e051c4ed-kube-api-access-q4lj9\") pod \"klusterlet-addon-workmgr-5d8896d549-vgrc4\" (UID: \"29817449-b2c8-40eb-9db7-c3d0e051c4ed\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8896d549-vgrc4"
Jul 28 17:10:09.367367 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.367337    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1891b513-8781-471c-a563-00e9fa143d76-image-registry-private-configuration\") pod \"image-registry-fb6888f7-cwxc9\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") " pod="openshift-image-registry/image-registry-fb6888f7-cwxc9"
Jul 28 17:10:09.367516 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.367366    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1891b513-8781-471c-a563-00e9fa143d76-trusted-ca\") pod \"image-registry-fb6888f7-cwxc9\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") " pod="openshift-image-registry/image-registry-fb6888f7-cwxc9"
Jul 28 17:10:09.367516 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.367392    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6f9f5b85-a06a-489f-a8dd-3408da0a5e8a-hub\") pod \"cluster-proxy-proxy-agent-d9b97dbc5-v2nc6\" (UID: \"6f9f5b85-a06a-489f-a8dd-3408da0a5e8a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9b97dbc5-v2nc6"
Jul 28 17:10:09.367516 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.367459    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1891b513-8781-471c-a563-00e9fa143d76-registry-tls\") pod \"image-registry-fb6888f7-cwxc9\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") " pod="openshift-image-registry/image-registry-fb6888f7-cwxc9"
Jul 28 17:10:09.367659 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.367506    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/496b869a-13ac-4171-8068-7c839c75b1e9-stats-auth\") pod \"router-default-5555ff8d6c-6mzsx\" (UID: \"496b869a-13ac-4171-8068-7c839c75b1e9\") " pod="openshift-ingress/router-default-5555ff8d6c-6mzsx"
Jul 28 17:10:09.367659 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.367545    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6f9f5b85-a06a-489f-a8dd-3408da0a5e8a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d9b97dbc5-v2nc6\" (UID: \"6f9f5b85-a06a-489f-a8dd-3408da0a5e8a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9b97dbc5-v2nc6"
Jul 28 17:10:09.367659 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.367586    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1891b513-8781-471c-a563-00e9fa143d76-ca-trust-extracted\") pod \"image-registry-fb6888f7-cwxc9\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") " pod="openshift-image-registry/image-registry-fb6888f7-cwxc9"
Jul 28 17:10:09.367659 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.367620    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1891b513-8781-471c-a563-00e9fa143d76-registry-certificates\") pod \"image-registry-fb6888f7-cwxc9\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") " pod="openshift-image-registry/image-registry-fb6888f7-cwxc9"
Jul 28 17:10:09.367659 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.367648    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1891b513-8781-471c-a563-00e9fa143d76-installation-pull-secrets\") pod \"image-registry-fb6888f7-cwxc9\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") " pod="openshift-image-registry/image-registry-fb6888f7-cwxc9"
Jul 28 17:10:09.367923 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.367672    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/496b869a-13ac-4171-8068-7c839c75b1e9-service-ca-bundle\") pod \"router-default-5555ff8d6c-6mzsx\" (UID: \"496b869a-13ac-4171-8068-7c839c75b1e9\") " pod="openshift-ingress/router-default-5555ff8d6c-6mzsx"
Jul 28 17:10:09.367923 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.367698    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh45t\" (UniqueName: \"kubernetes.io/projected/1891b513-8781-471c-a563-00e9fa143d76-kube-api-access-qh45t\") pod \"image-registry-fb6888f7-cwxc9\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") " pod="openshift-image-registry/image-registry-fb6888f7-cwxc9"
Jul 28 17:10:09.367923 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.367772    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p644g\" (UniqueName: \"kubernetes.io/projected/6f9f5b85-a06a-489f-a8dd-3408da0a5e8a-kube-api-access-p644g\") pod \"cluster-proxy-proxy-agent-d9b97dbc5-v2nc6\" (UID: \"6f9f5b85-a06a-489f-a8dd-3408da0a5e8a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9b97dbc5-v2nc6"
Jul 28 17:10:09.367923 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.367804    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29817449-b2c8-40eb-9db7-c3d0e051c4ed-tmp\") pod \"klusterlet-addon-workmgr-5d8896d549-vgrc4\" (UID: \"29817449-b2c8-40eb-9db7-c3d0e051c4ed\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8896d549-vgrc4"
Jul 28 17:10:09.367923 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.367827    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6f9f5b85-a06a-489f-a8dd-3408da0a5e8a-ca\") pod \"cluster-proxy-proxy-agent-d9b97dbc5-v2nc6\" (UID: \"6f9f5b85-a06a-489f-a8dd-3408da0a5e8a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9b97dbc5-v2nc6"
Jul 28 17:10:09.367923 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.367853    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1891b513-8781-471c-a563-00e9fa143d76-bound-sa-token\") pod \"image-registry-fb6888f7-cwxc9\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") " pod="openshift-image-registry/image-registry-fb6888f7-cwxc9"
Jul 28 17:10:09.367923 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.367881    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/496b869a-13ac-4171-8068-7c839c75b1e9-metrics-certs\") pod \"router-default-5555ff8d6c-6mzsx\" (UID: \"496b869a-13ac-4171-8068-7c839c75b1e9\") " pod="openshift-ingress/router-default-5555ff8d6c-6mzsx"
Jul 28 17:10:09.368246 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.368085    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f9f5b85-a06a-489f-a8dd-3408da0a5e8a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d9b97dbc5-v2nc6\" (UID: \"6f9f5b85-a06a-489f-a8dd-3408da0a5e8a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9b97dbc5-v2nc6"
Jul 28 17:10:09.368313 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.368285    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29817449-b2c8-40eb-9db7-c3d0e051c4ed-tmp\") pod \"klusterlet-addon-workmgr-5d8896d549-vgrc4\" (UID: \"29817449-b2c8-40eb-9db7-c3d0e051c4ed\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8896d549-vgrc4"
Jul 28 17:10:09.370044 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.370021    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6f9f5b85-a06a-489f-a8dd-3408da0a5e8a-hub\") pod \"cluster-proxy-proxy-agent-d9b97dbc5-v2nc6\" (UID: \"6f9f5b85-a06a-489f-a8dd-3408da0a5e8a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9b97dbc5-v2nc6"
Jul 28 17:10:09.370201 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.370058    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/29817449-b2c8-40eb-9db7-c3d0e051c4ed-klusterlet-config\") pod \"klusterlet-addon-workmgr-5d8896d549-vgrc4\" (UID: \"29817449-b2c8-40eb-9db7-c3d0e051c4ed\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8896d549-vgrc4"
Jul 28 17:10:09.370201 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.370171    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f9f5b85-a06a-489f-a8dd-3408da0a5e8a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d9b97dbc5-v2nc6\" (UID: \"6f9f5b85-a06a-489f-a8dd-3408da0a5e8a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9b97dbc5-v2nc6"
Jul 28 17:10:09.370314 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.370243    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6f9f5b85-a06a-489f-a8dd-3408da0a5e8a-ca\") pod \"cluster-proxy-proxy-agent-d9b97dbc5-v2nc6\" (UID: \"6f9f5b85-a06a-489f-a8dd-3408da0a5e8a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9b97dbc5-v2nc6"
Jul 28 17:10:09.370368 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.370344    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6f9f5b85-a06a-489f-a8dd-3408da0a5e8a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d9b97dbc5-v2nc6\" (UID: \"6f9f5b85-a06a-489f-a8dd-3408da0a5e8a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9b97dbc5-v2nc6"
Jul 28 17:10:09.405420 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.405399    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p644g\" (UniqueName: \"kubernetes.io/projected/6f9f5b85-a06a-489f-a8dd-3408da0a5e8a-kube-api-access-p644g\") pod \"cluster-proxy-proxy-agent-d9b97dbc5-v2nc6\" (UID: \"6f9f5b85-a06a-489f-a8dd-3408da0a5e8a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9b97dbc5-v2nc6"
Jul 28 17:10:09.408782 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.408721    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4lj9\" (UniqueName: \"kubernetes.io/projected/29817449-b2c8-40eb-9db7-c3d0e051c4ed-kube-api-access-q4lj9\") pod \"klusterlet-addon-workmgr-5d8896d549-vgrc4\" (UID: \"29817449-b2c8-40eb-9db7-c3d0e051c4ed\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8896d549-vgrc4"
Jul 28 17:10:09.448878 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.448853    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:09.469095 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.469069    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1891b513-8781-471c-a563-00e9fa143d76-image-registry-private-configuration\") pod \"image-registry-fb6888f7-cwxc9\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") " pod="openshift-image-registry/image-registry-fb6888f7-cwxc9"
Jul 28 17:10:09.469213 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.469099    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1891b513-8781-471c-a563-00e9fa143d76-trusted-ca\") pod \"image-registry-fb6888f7-cwxc9\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") " pod="openshift-image-registry/image-registry-fb6888f7-cwxc9"
Jul 28 17:10:09.469213 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.469128    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1891b513-8781-471c-a563-00e9fa143d76-registry-tls\") pod \"image-registry-fb6888f7-cwxc9\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") " pod="openshift-image-registry/image-registry-fb6888f7-cwxc9"
Jul 28 17:10:09.469213 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.469145    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/496b869a-13ac-4171-8068-7c839c75b1e9-stats-auth\") pod \"router-default-5555ff8d6c-6mzsx\" (UID: \"496b869a-13ac-4171-8068-7c839c75b1e9\") " pod="openshift-ingress/router-default-5555ff8d6c-6mzsx"
Jul 28 17:10:09.469213 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.469179    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1891b513-8781-471c-a563-00e9fa143d76-ca-trust-extracted\") pod \"image-registry-fb6888f7-cwxc9\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") " pod="openshift-image-registry/image-registry-fb6888f7-cwxc9"
Jul 28 17:10:09.469213 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.469195    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1891b513-8781-471c-a563-00e9fa143d76-registry-certificates\") pod \"image-registry-fb6888f7-cwxc9\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") " pod="openshift-image-registry/image-registry-fb6888f7-cwxc9"
Jul 28 17:10:09.469451 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.469219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1891b513-8781-471c-a563-00e9fa143d76-installation-pull-secrets\") pod \"image-registry-fb6888f7-cwxc9\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") " pod="openshift-image-registry/image-registry-fb6888f7-cwxc9"
Jul 28 17:10:09.470584 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.469674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/496b869a-13ac-4171-8068-7c839c75b1e9-service-ca-bundle\") pod \"router-default-5555ff8d6c-6mzsx\" (UID: \"496b869a-13ac-4171-8068-7c839c75b1e9\") " pod="openshift-ingress/router-default-5555ff8d6c-6mzsx"
Jul 28 17:10:09.470584 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.469732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh45t\" (UniqueName: \"kubernetes.io/projected/1891b513-8781-471c-a563-00e9fa143d76-kube-api-access-qh45t\") pod \"image-registry-fb6888f7-cwxc9\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") " pod="openshift-image-registry/image-registry-fb6888f7-cwxc9"
Jul 28 17:10:09.470584 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.469775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1891b513-8781-471c-a563-00e9fa143d76-bound-sa-token\") pod \"image-registry-fb6888f7-cwxc9\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") " pod="openshift-image-registry/image-registry-fb6888f7-cwxc9"
Jul 28 17:10:09.470584 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.469805    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/496b869a-13ac-4171-8068-7c839c75b1e9-metrics-certs\") pod \"router-default-5555ff8d6c-6mzsx\" (UID: \"496b869a-13ac-4171-8068-7c839c75b1e9\") " pod="openshift-ingress/router-default-5555ff8d6c-6mzsx"
Jul 28 17:10:09.470584 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.469858    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/496b869a-13ac-4171-8068-7c839c75b1e9-default-certificate\") pod \"router-default-5555ff8d6c-6mzsx\" (UID: \"496b869a-13ac-4171-8068-7c839c75b1e9\") " pod="openshift-ingress/router-default-5555ff8d6c-6mzsx"
Jul 28 17:10:09.470584 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.469903    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzsgr\" (UniqueName: \"kubernetes.io/projected/496b869a-13ac-4171-8068-7c839c75b1e9-kube-api-access-lzsgr\") pod \"router-default-5555ff8d6c-6mzsx\" (UID: \"496b869a-13ac-4171-8068-7c839c75b1e9\") " pod="openshift-ingress/router-default-5555ff8d6c-6mzsx"
Jul 28 17:10:09.470584 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.470218    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1891b513-8781-471c-a563-00e9fa143d76-registry-certificates\") pod \"image-registry-fb6888f7-cwxc9\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") " pod="openshift-image-registry/image-registry-fb6888f7-cwxc9"
Jul 28 17:10:09.470584 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.469690    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1891b513-8781-471c-a563-00e9fa143d76-ca-trust-extracted\") pod \"image-registry-fb6888f7-cwxc9\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") " pod="openshift-image-registry/image-registry-fb6888f7-cwxc9"
Jul 28 17:10:09.470584 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.470356    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/496b869a-13ac-4171-8068-7c839c75b1e9-service-ca-bundle\") pod \"router-default-5555ff8d6c-6mzsx\" (UID: \"496b869a-13ac-4171-8068-7c839c75b1e9\") " pod="openshift-ingress/router-default-5555ff8d6c-6mzsx"
Jul 28 17:10:09.470584 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.470500    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1891b513-8781-471c-a563-00e9fa143d76-trusted-ca\") pod \"image-registry-fb6888f7-cwxc9\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") " pod="openshift-image-registry/image-registry-fb6888f7-cwxc9"
Jul 28 17:10:09.471725 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.471704    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1891b513-8781-471c-a563-00e9fa143d76-registry-tls\") pod \"image-registry-fb6888f7-cwxc9\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") " pod="openshift-image-registry/image-registry-fb6888f7-cwxc9"
Jul 28 17:10:09.471900 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.471879    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1891b513-8781-471c-a563-00e9fa143d76-image-registry-private-configuration\") pod \"image-registry-fb6888f7-cwxc9\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") " pod="openshift-image-registry/image-registry-fb6888f7-cwxc9"
Jul 28 17:10:09.472025 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.472010    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8896d549-vgrc4"
Jul 28 17:10:09.472092 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.472021    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1891b513-8781-471c-a563-00e9fa143d76-installation-pull-secrets\") pod \"image-registry-fb6888f7-cwxc9\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") " pod="openshift-image-registry/image-registry-fb6888f7-cwxc9"
Jul 28 17:10:09.472472 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.472426    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/496b869a-13ac-4171-8068-7c839c75b1e9-stats-auth\") pod \"router-default-5555ff8d6c-6mzsx\" (UID: \"496b869a-13ac-4171-8068-7c839c75b1e9\") " pod="openshift-ingress/router-default-5555ff8d6c-6mzsx"
Jul 28 17:10:09.472767 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.472748    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/496b869a-13ac-4171-8068-7c839c75b1e9-default-certificate\") pod \"router-default-5555ff8d6c-6mzsx\" (UID: \"496b869a-13ac-4171-8068-7c839c75b1e9\") " pod="openshift-ingress/router-default-5555ff8d6c-6mzsx"
Jul 28 17:10:09.473141 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.473125    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/496b869a-13ac-4171-8068-7c839c75b1e9-metrics-certs\") pod \"router-default-5555ff8d6c-6mzsx\" (UID: \"496b869a-13ac-4171-8068-7c839c75b1e9\") " pod="openshift-ingress/router-default-5555ff8d6c-6mzsx"
Jul 28 17:10:09.479777 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.479745    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh45t\" (UniqueName: \"kubernetes.io/projected/1891b513-8781-471c-a563-00e9fa143d76-kube-api-access-qh45t\") pod \"image-registry-fb6888f7-cwxc9\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") " pod="openshift-image-registry/image-registry-fb6888f7-cwxc9"
Jul 28 17:10:09.479859 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.479821    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1891b513-8781-471c-a563-00e9fa143d76-bound-sa-token\") pod \"image-registry-fb6888f7-cwxc9\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") " pod="openshift-image-registry/image-registry-fb6888f7-cwxc9"
Jul 28 17:10:09.479990 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.479975    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzsgr\" (UniqueName: \"kubernetes.io/projected/496b869a-13ac-4171-8068-7c839c75b1e9-kube-api-access-lzsgr\") pod \"router-default-5555ff8d6c-6mzsx\" (UID: \"496b869a-13ac-4171-8068-7c839c75b1e9\") " pod="openshift-ingress/router-default-5555ff8d6c-6mzsx"
Jul 28 17:10:09.486615 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.486599    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9b97dbc5-v2nc6"
Jul 28 17:10:09.527755 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.526517    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb6888f7-cwxc9"
Jul 28 17:10:09.535642 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.535333    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb6888f7-cwxc9"
Jul 28 17:10:09.563303 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.563273    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:09.570287 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.570265    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1891b513-8781-471c-a563-00e9fa143d76-installation-pull-secrets\") pod \"1891b513-8781-471c-a563-00e9fa143d76\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") "
Jul 28 17:10:09.570384 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.570304    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1891b513-8781-471c-a563-00e9fa143d76-trusted-ca\") pod \"1891b513-8781-471c-a563-00e9fa143d76\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") "
Jul 28 17:10:09.570384 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.570327    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1891b513-8781-471c-a563-00e9fa143d76-image-registry-private-configuration\") pod \"1891b513-8781-471c-a563-00e9fa143d76\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") "
Jul 28 17:10:09.570384 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.570348    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qh45t\" (UniqueName: \"kubernetes.io/projected/1891b513-8781-471c-a563-00e9fa143d76-kube-api-access-qh45t\") pod \"1891b513-8781-471c-a563-00e9fa143d76\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") "
Jul 28 17:10:09.570384 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.570375    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1891b513-8781-471c-a563-00e9fa143d76-ca-trust-extracted\") pod \"1891b513-8781-471c-a563-00e9fa143d76\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") "
Jul 28 17:10:09.570541 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.570392    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1891b513-8781-471c-a563-00e9fa143d76-registry-certificates\") pod \"1891b513-8781-471c-a563-00e9fa143d76\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") "
Jul 28 17:10:09.570541 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.570414    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1891b513-8781-471c-a563-00e9fa143d76-bound-sa-token\") pod \"1891b513-8781-471c-a563-00e9fa143d76\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") "
Jul 28 17:10:09.570541 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.570444    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1891b513-8781-471c-a563-00e9fa143d76-registry-tls\") pod \"1891b513-8781-471c-a563-00e9fa143d76\" (UID: \"1891b513-8781-471c-a563-00e9fa143d76\") "
Jul 28 17:10:09.571565 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.571018    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1891b513-8781-471c-a563-00e9fa143d76-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1891b513-8781-471c-a563-00e9fa143d76" (UID: "1891b513-8781-471c-a563-00e9fa143d76"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:10:09.571565 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.571306    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1891b513-8781-471c-a563-00e9fa143d76-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1891b513-8781-471c-a563-00e9fa143d76" (UID: "1891b513-8781-471c-a563-00e9fa143d76"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:10:09.571832 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.571690    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5555ff8d6c-6mzsx"
Jul 28 17:10:09.572240 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.572211    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1891b513-8781-471c-a563-00e9fa143d76-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1891b513-8781-471c-a563-00e9fa143d76" (UID: "1891b513-8781-471c-a563-00e9fa143d76"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:10:09.573419 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.573373    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1891b513-8781-471c-a563-00e9fa143d76-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1891b513-8781-471c-a563-00e9fa143d76" (UID: "1891b513-8781-471c-a563-00e9fa143d76"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:10:09.575852 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.575300    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1891b513-8781-471c-a563-00e9fa143d76-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1891b513-8781-471c-a563-00e9fa143d76" (UID: "1891b513-8781-471c-a563-00e9fa143d76"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:10:09.575852 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.575633    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1891b513-8781-471c-a563-00e9fa143d76-kube-api-access-qh45t" (OuterVolumeSpecName: "kube-api-access-qh45t") pod "1891b513-8781-471c-a563-00e9fa143d76" (UID: "1891b513-8781-471c-a563-00e9fa143d76"). InnerVolumeSpecName "kube-api-access-qh45t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:10:09.575852 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.575713    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1891b513-8781-471c-a563-00e9fa143d76-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1891b513-8781-471c-a563-00e9fa143d76" (UID: "1891b513-8781-471c-a563-00e9fa143d76"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:10:09.576088 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.576011    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1891b513-8781-471c-a563-00e9fa143d76-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1891b513-8781-471c-a563-00e9fa143d76" (UID: "1891b513-8781-471c-a563-00e9fa143d76"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:10:09.620374 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.619510    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8896d549-vgrc4"]
Jul 28 17:10:09.622719 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:10:09.622661    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29817449_b2c8_40eb_9db7_c3d0e051c4ed.slice/crio-13418f77de6541fbc7f90570d6345042ea3de46778d8b50994771166553e3960 WatchSource:0}: Error finding container 13418f77de6541fbc7f90570d6345042ea3de46778d8b50994771166553e3960: Status 404 returned error can't find the container with id 13418f77de6541fbc7f90570d6345042ea3de46778d8b50994771166553e3960
Jul 28 17:10:09.640195 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.640172    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9b97dbc5-v2nc6"]
Jul 28 17:10:09.642750 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:10:09.642722    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f9f5b85_a06a_489f_a8dd_3408da0a5e8a.slice/crio-9fbc257fee55d84174b9bc0d4f9b22603a2397f82b9f0d2939d2a2ea47b2b631 WatchSource:0}: Error finding container 9fbc257fee55d84174b9bc0d4f9b22603a2397f82b9f0d2939d2a2ea47b2b631: Status 404 returned error can't find the container with id 9fbc257fee55d84174b9bc0d4f9b22603a2397f82b9f0d2939d2a2ea47b2b631
Jul 28 17:10:09.653173 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.653101    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:09.671489 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.671433    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1891b513-8781-471c-a563-00e9fa143d76-registry-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:10:09.671489 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.671456    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1891b513-8781-471c-a563-00e9fa143d76-installation-pull-secrets\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:10:09.671489 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.671469    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1891b513-8781-471c-a563-00e9fa143d76-trusted-ca\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:10:09.671489 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.671483    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1891b513-8781-471c-a563-00e9fa143d76-image-registry-private-configuration\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:10:09.671685 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.671498    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh45t\" (UniqueName: \"kubernetes.io/projected/1891b513-8781-471c-a563-00e9fa143d76-kube-api-access-qh45t\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:10:09.671685 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.671510    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1891b513-8781-471c-a563-00e9fa143d76-ca-trust-extracted\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:10:09.671685 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.671521    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1891b513-8781-471c-a563-00e9fa143d76-registry-certificates\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:10:09.671685 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.671533    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1891b513-8781-471c-a563-00e9fa143d76-bound-sa-token\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:10:09.700573 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.700539    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5555ff8d6c-6mzsx"]
Jul 28 17:10:09.704775 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:10:09.704752    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496b869a_13ac_4171_8068_7c839c75b1e9.slice/crio-4f0a9508ac6c5357d78c051a370f091d1fd70c16737a57738bbed03c1b3c506c WatchSource:0}: Error finding container 4f0a9508ac6c5357d78c051a370f091d1fd70c16737a57738bbed03c1b3c506c: Status 404 returned error can't find the container with id 4f0a9508ac6c5357d78c051a370f091d1fd70c16737a57738bbed03c1b3c506c
Jul 28 17:10:09.755315 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:09.755294    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:10.119482 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:10.119401    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:10.222051 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:10.222017    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:10.342078 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:10.342045    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:10.430751 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:10.430715    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:10.532830 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:10.532775    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9b97dbc5-v2nc6" event={"ID":"6f9f5b85-a06a-489f-a8dd-3408da0a5e8a","Type":"ContainerStarted","Data":"9fbc257fee55d84174b9bc0d4f9b22603a2397f82b9f0d2939d2a2ea47b2b631"}
Jul 28 17:10:10.534700 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:10.534667    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8896d549-vgrc4" event={"ID":"29817449-b2c8-40eb-9db7-c3d0e051c4ed","Type":"ContainerStarted","Data":"13418f77de6541fbc7f90570d6345042ea3de46778d8b50994771166553e3960"}
Jul 28 17:10:10.536368 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:10.536345    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb6888f7-cwxc9"
Jul 28 17:10:10.537495 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:10.536861    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5555ff8d6c-6mzsx" event={"ID":"496b869a-13ac-4171-8068-7c839c75b1e9","Type":"ContainerStarted","Data":"b90129471b59cb25d2e5e34e9c2d509e39bde96cc6edb874630469b1cefe7251"}
Jul 28 17:10:10.537495 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:10.536910    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5555ff8d6c-6mzsx" event={"ID":"496b869a-13ac-4171-8068-7c839c75b1e9","Type":"ContainerStarted","Data":"4f0a9508ac6c5357d78c051a370f091d1fd70c16737a57738bbed03c1b3c506c"}
Jul 28 17:10:10.564214 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:10.564166    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5555ff8d6c-6mzsx" podStartSLOduration=1.56414937 podStartE2EDuration="1.56414937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:10.564060718 +0000 UTC m=+58.838677151" watchObservedRunningTime="2026-07-28 17:10:10.56414937 +0000 UTC m=+58.838765802"
Jul 28 17:10:10.572674 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:10.572653    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5555ff8d6c-6mzsx"
Jul 28 17:10:10.575937 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:10.575917    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5555ff8d6c-6mzsx"
Jul 28 17:10:10.594926 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:10.594903    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fb6888f7-cwxc9"]
Jul 28 17:10:10.597563 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:10.597539    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-fb6888f7-cwxc9"]
Jul 28 17:10:10.604038 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:10.604013    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:10.770317 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:10.770268    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:11.453381 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:11.453338    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9l7dh"
Jul 28 17:10:11.542719 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:11.542687    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5555ff8d6c-6mzsx"
Jul 28 17:10:11.544625 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:11.544448    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5555ff8d6c-6mzsx"
Jul 28 17:10:11.770532 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:11.770449    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:11.878103 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:11.878068    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:12.149899 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:12.149864    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:12.250682 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:12.250652    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:12.290467 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:12.290439    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1891b513-8781-471c-a563-00e9fa143d76" path="/var/lib/kubelet/pods/1891b513-8781-471c-a563-00e9fa143d76/volumes"
Jul 28 17:10:12.570930 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:12.570836    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:12.640473 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:12.640416    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:15.557715 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:15.557675    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9b97dbc5-v2nc6" event={"ID":"6f9f5b85-a06a-489f-a8dd-3408da0a5e8a","Type":"ContainerStarted","Data":"0e386aa4c1f3dada6d60ba579104c040f3cdd2d5357a01dc194fbeda2a3bc25f"}
Jul 28 17:10:15.558161 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:15.557723    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9b97dbc5-v2nc6" event={"ID":"6f9f5b85-a06a-489f-a8dd-3408da0a5e8a","Type":"ContainerStarted","Data":"075f9c36bf30a38ed287191d289537e012976ac311de014552f778837e8c7ae8"}
Jul 28 17:10:15.558161 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:15.557746    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9b97dbc5-v2nc6" event={"ID":"6f9f5b85-a06a-489f-a8dd-3408da0a5e8a","Type":"ContainerStarted","Data":"6211780a3aa805647a6f925e78476d6c22dd3c722181eb035ea579a3a185bea5"}
Jul 28 17:10:15.559095 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:15.559070    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8896d549-vgrc4" event={"ID":"29817449-b2c8-40eb-9db7-c3d0e051c4ed","Type":"ContainerStarted","Data":"cebd2c3bb870cb42e20992103d85f4d88ff81af4025735d8b8eb415274bdc86b"}
Jul 28 17:10:15.559214 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:15.559197    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8896d549-vgrc4"
Jul 28 17:10:15.561391 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:15.561158    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8896d549-vgrc4"
Jul 28 17:10:15.578932 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:15.578890    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9b97dbc5-v2nc6" podStartSLOduration=1.445748859 podStartE2EDuration="6.578878097s" totalImagesPullingTime="5.133129238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:15.576886119 +0000 UTC m=+63.851502551" watchObservedRunningTime="2026-07-28 17:10:15.578878097 +0000 UTC m=+63.853494507"
Jul 28 17:10:15.594087 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:15.594050    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5d8896d549-vgrc4" podStartSLOduration=1.443621979 podStartE2EDuration="6.594038159s" totalImagesPullingTime="5.15041618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:15.593300797 +0000 UTC m=+63.867917229" watchObservedRunningTime="2026-07-28 17:10:15.594038159 +0000 UTC m=+63.868654590"
Jul 28 17:10:16.286292 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:16.286257    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:16.359591 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:16.359558    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:16.521834 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:16.521801    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mvdj9"
Jul 28 17:10:17.544805 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:17.544773    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:17.646028 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:17.645990    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:17.737496 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:17.737465    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:17.890482 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:17.890400    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:17.935397 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:17.935370    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgzn7\" (UniqueName: \"kubernetes.io/projected/33068506-3820-4d64-bc6b-33ea180dca70-kube-api-access-jgzn7\") pod \"network-check-target-2nt29\" (UID: \"33068506-3820-4d64-bc6b-33ea180dca70\") " pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:10:17.935534 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:17.935422    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a7b47e4-993e-4527-a00f-4ebbbc050348-metrics-certs\") pod \"network-metrics-daemon-zkf8x\" (UID: \"9a7b47e4-993e-4527-a00f-4ebbbc050348\") " pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:10:17.935534 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:17.935457    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/189dbf08-647d-42a8-975f-39838ad4edf9-original-pull-secret\") pod \"global-pull-secret-syncer-jwmpm\" (UID: \"189dbf08-647d-42a8-975f-39838ad4edf9\") " pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:10:17.938703 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:17.938685    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:10:17.939794 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:17.939772    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:10:17.939877 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:17.939807    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:10:17.948787 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:17.948765    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/189dbf08-647d-42a8-975f-39838ad4edf9-original-pull-secret\") pod \"global-pull-secret-syncer-jwmpm\" (UID: \"189dbf08-647d-42a8-975f-39838ad4edf9\") " pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:10:17.948848 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:17.948807    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a7b47e4-993e-4527-a00f-4ebbbc050348-metrics-certs\") pod \"network-metrics-daemon-zkf8x\" (UID: \"9a7b47e4-993e-4527-a00f-4ebbbc050348\") " pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:10:17.949785 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:17.949770    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:10:17.958904 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:17.958881    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgzn7\" (UniqueName: \"kubernetes.io/projected/33068506-3820-4d64-bc6b-33ea180dca70-kube-api-access-jgzn7\") pod \"network-check-target-2nt29\" (UID: \"33068506-3820-4d64-bc6b-33ea180dca70\") " pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:10:18.201159 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.201131    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7zr5q\""
Jul 28 17:10:18.205636 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.205620    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwmpm"
Jul 28 17:10:18.208736 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.208720    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zkf8x"
Jul 28 17:10:18.216698 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.216680    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b4dt6\""
Jul 28 17:10:18.222170 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.222146    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:10:18.228084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.228056    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:18.299578 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.299549    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:18.360393 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.360366    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zkf8x"]
Jul 28 17:10:18.363261 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:10:18.363235    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7b47e4_993e_4527_a00f_4ebbbc050348.slice/crio-4f9ebfb5dc2ef2b97b5988ae6f116f6f1908dd985fdd2957bb1a21c7a0a41adb WatchSource:0}: Error finding container 4f9ebfb5dc2ef2b97b5988ae6f116f6f1908dd985fdd2957bb1a21c7a0a41adb: Status 404 returned error can't find the container with id 4f9ebfb5dc2ef2b97b5988ae6f116f6f1908dd985fdd2957bb1a21c7a0a41adb
Jul 28 17:10:18.536974 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.536866    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-z65fp"]
Jul 28 17:10:18.571777 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.571749    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z65fp"]
Jul 28 17:10:18.571777 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.571787    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zkf8x" event={"ID":"9a7b47e4-993e-4527-a00f-4ebbbc050348","Type":"ContainerStarted","Data":"4f9ebfb5dc2ef2b97b5988ae6f116f6f1908dd985fdd2957bb1a21c7a0a41adb"}
Jul 28 17:10:18.572282 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.571891    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z65fp"
Jul 28 17:10:18.575085 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.575062    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 17:10:18.575198 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.575105    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 17:10:18.575479 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.575444    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fs9nh\""
Jul 28 17:10:18.575595 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.575579    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 17:10:18.576518 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.576499    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 17:10:18.589635 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.589612    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:18.594868 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.594847    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jwmpm"]
Jul 28 17:10:18.596623 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.596608    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2nt29"]
Jul 28 17:10:18.598364 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:10:18.598311    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189dbf08_647d_42a8_975f_39838ad4edf9.slice/crio-cf5f6a5a476d05b0d5b11d108e6c7e357b23d88f8ba5a4d534df6edf59b1efca WatchSource:0}: Error finding container cf5f6a5a476d05b0d5b11d108e6c7e357b23d88f8ba5a4d534df6edf59b1efca: Status 404 returned error can't find the container with id cf5f6a5a476d05b0d5b11d108e6c7e357b23d88f8ba5a4d534df6edf59b1efca
Jul 28 17:10:18.600477 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:10:18.600460    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33068506_3820_4d64_bc6b_33ea180dca70.slice/crio-3111b2e369dd321e9915063f488cfb1d6cc795147001ba0501150a828ce31e91 WatchSource:0}: Error finding container 3111b2e369dd321e9915063f488cfb1d6cc795147001ba0501150a828ce31e91: Status 404 returned error can't find the container with id 3111b2e369dd321e9915063f488cfb1d6cc795147001ba0501150a828ce31e91
Jul 28 17:10:18.641414 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.641385    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pptgm\" (UniqueName: \"kubernetes.io/projected/137885a5-ec67-4cfd-baa1-2a43a9389bb1-kube-api-access-pptgm\") pod \"insights-runtime-extractor-z65fp\" (UID: \"137885a5-ec67-4cfd-baa1-2a43a9389bb1\") " pod="openshift-insights/insights-runtime-extractor-z65fp"
Jul 28 17:10:18.641537 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.641430    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/137885a5-ec67-4cfd-baa1-2a43a9389bb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z65fp\" (UID: \"137885a5-ec67-4cfd-baa1-2a43a9389bb1\") " pod="openshift-insights/insights-runtime-extractor-z65fp"
Jul 28 17:10:18.641537 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.641450    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/137885a5-ec67-4cfd-baa1-2a43a9389bb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z65fp\" (UID: \"137885a5-ec67-4cfd-baa1-2a43a9389bb1\") " pod="openshift-insights/insights-runtime-extractor-z65fp"
Jul 28 17:10:18.641537 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.641514    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/137885a5-ec67-4cfd-baa1-2a43a9389bb1-data-volume\") pod \"insights-runtime-extractor-z65fp\" (UID: \"137885a5-ec67-4cfd-baa1-2a43a9389bb1\") " pod="openshift-insights/insights-runtime-extractor-z65fp"
Jul 28 17:10:18.641644 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.641566    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/137885a5-ec67-4cfd-baa1-2a43a9389bb1-crio-socket\") pod \"insights-runtime-extractor-z65fp\" (UID: \"137885a5-ec67-4cfd-baa1-2a43a9389bb1\") " pod="openshift-insights/insights-runtime-extractor-z65fp"
Jul 28 17:10:18.712859 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.712831    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:18.742503 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.742471    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/137885a5-ec67-4cfd-baa1-2a43a9389bb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z65fp\" (UID: \"137885a5-ec67-4cfd-baa1-2a43a9389bb1\") " pod="openshift-insights/insights-runtime-extractor-z65fp"
Jul 28 17:10:18.742503 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.742504    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/137885a5-ec67-4cfd-baa1-2a43a9389bb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z65fp\" (UID: \"137885a5-ec67-4cfd-baa1-2a43a9389bb1\") " pod="openshift-insights/insights-runtime-extractor-z65fp"
Jul 28 17:10:18.742675 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.742535    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/137885a5-ec67-4cfd-baa1-2a43a9389bb1-data-volume\") pod \"insights-runtime-extractor-z65fp\" (UID: \"137885a5-ec67-4cfd-baa1-2a43a9389bb1\") " pod="openshift-insights/insights-runtime-extractor-z65fp"
Jul 28 17:10:18.742675 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.742571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/137885a5-ec67-4cfd-baa1-2a43a9389bb1-crio-socket\") pod \"insights-runtime-extractor-z65fp\" (UID: \"137885a5-ec67-4cfd-baa1-2a43a9389bb1\") " pod="openshift-insights/insights-runtime-extractor-z65fp"
Jul 28 17:10:18.742675 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.742601    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pptgm\" (UniqueName: \"kubernetes.io/projected/137885a5-ec67-4cfd-baa1-2a43a9389bb1-kube-api-access-pptgm\") pod \"insights-runtime-extractor-z65fp\" (UID: \"137885a5-ec67-4cfd-baa1-2a43a9389bb1\") " pod="openshift-insights/insights-runtime-extractor-z65fp"
Jul 28 17:10:18.742675 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:10:18.742608    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:10:18.742675 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:10:18.742630    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:10:18.742675 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.742666    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/137885a5-ec67-4cfd-baa1-2a43a9389bb1-crio-socket\") pod \"insights-runtime-extractor-z65fp\" (UID: \"137885a5-ec67-4cfd-baa1-2a43a9389bb1\") " pod="openshift-insights/insights-runtime-extractor-z65fp"
Jul 28 17:10:18.742873 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:10:18.742682    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/137885a5-ec67-4cfd-baa1-2a43a9389bb1-kube-rbac-proxy-cm podName:137885a5-ec67-4cfd-baa1-2a43a9389bb1 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:19.242661442 +0000 UTC m=+67.517277871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/137885a5-ec67-4cfd-baa1-2a43a9389bb1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z65fp" (UID: "137885a5-ec67-4cfd-baa1-2a43a9389bb1") : configmap "kube-rbac-proxy" not found
Jul 28 17:10:18.742873 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:10:18.742701    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/137885a5-ec67-4cfd-baa1-2a43a9389bb1-insights-runtime-extractor-tls podName:137885a5-ec67-4cfd-baa1-2a43a9389bb1 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:19.242692466 +0000 UTC m=+67.517308877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/137885a5-ec67-4cfd-baa1-2a43a9389bb1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z65fp" (UID: "137885a5-ec67-4cfd-baa1-2a43a9389bb1") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:10:18.742988 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.742872    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/137885a5-ec67-4cfd-baa1-2a43a9389bb1-data-volume\") pod \"insights-runtime-extractor-z65fp\" (UID: \"137885a5-ec67-4cfd-baa1-2a43a9389bb1\") " pod="openshift-insights/insights-runtime-extractor-z65fp"
Jul 28 17:10:18.753826 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.753806    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pptgm\" (UniqueName: \"kubernetes.io/projected/137885a5-ec67-4cfd-baa1-2a43a9389bb1-kube-api-access-pptgm\") pod \"insights-runtime-extractor-z65fp\" (UID: \"137885a5-ec67-4cfd-baa1-2a43a9389bb1\") " pod="openshift-insights/insights-runtime-extractor-z65fp"
Jul 28 17:10:18.879589 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.879524    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:18.941291 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:18.941263    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:19.246465 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:19.246429    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/137885a5-ec67-4cfd-baa1-2a43a9389bb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z65fp\" (UID: \"137885a5-ec67-4cfd-baa1-2a43a9389bb1\") " pod="openshift-insights/insights-runtime-extractor-z65fp"
Jul 28 17:10:19.246465 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:19.246466    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/137885a5-ec67-4cfd-baa1-2a43a9389bb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z65fp\" (UID: \"137885a5-ec67-4cfd-baa1-2a43a9389bb1\") " pod="openshift-insights/insights-runtime-extractor-z65fp"
Jul 28 17:10:19.246665 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:10:19.246578    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:10:19.246665 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:10:19.246639    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/137885a5-ec67-4cfd-baa1-2a43a9389bb1-insights-runtime-extractor-tls podName:137885a5-ec67-4cfd-baa1-2a43a9389bb1 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:20.246624619 +0000 UTC m=+68.521241033 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/137885a5-ec67-4cfd-baa1-2a43a9389bb1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z65fp" (UID: "137885a5-ec67-4cfd-baa1-2a43a9389bb1") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:10:19.246737 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:10:19.246577    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:10:19.246737 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:10:19.246705    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/137885a5-ec67-4cfd-baa1-2a43a9389bb1-kube-rbac-proxy-cm podName:137885a5-ec67-4cfd-baa1-2a43a9389bb1 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:20.2466934 +0000 UTC m=+68.521309825 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/137885a5-ec67-4cfd-baa1-2a43a9389bb1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z65fp" (UID: "137885a5-ec67-4cfd-baa1-2a43a9389bb1") : configmap "kube-rbac-proxy" not found
Jul 28 17:10:19.577433 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:19.577355    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jwmpm" event={"ID":"189dbf08-647d-42a8-975f-39838ad4edf9","Type":"ContainerStarted","Data":"cf5f6a5a476d05b0d5b11d108e6c7e357b23d88f8ba5a4d534df6edf59b1efca"}
Jul 28 17:10:19.578670 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:19.578637    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2nt29" event={"ID":"33068506-3820-4d64-bc6b-33ea180dca70","Type":"ContainerStarted","Data":"3111b2e369dd321e9915063f488cfb1d6cc795147001ba0501150a828ce31e91"}
Jul 28 17:10:20.255893 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:20.255838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/137885a5-ec67-4cfd-baa1-2a43a9389bb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z65fp\" (UID: \"137885a5-ec67-4cfd-baa1-2a43a9389bb1\") " pod="openshift-insights/insights-runtime-extractor-z65fp"
Jul 28 17:10:20.255893 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:20.255889    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/137885a5-ec67-4cfd-baa1-2a43a9389bb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z65fp\" (UID: \"137885a5-ec67-4cfd-baa1-2a43a9389bb1\") " pod="openshift-insights/insights-runtime-extractor-z65fp"
Jul 28 17:10:20.256219 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:10:20.256006    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:10:20.256219 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:10:20.256029    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:10:20.256219 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:10:20.256076    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/137885a5-ec67-4cfd-baa1-2a43a9389bb1-kube-rbac-proxy-cm podName:137885a5-ec67-4cfd-baa1-2a43a9389bb1 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:22.256058222 +0000 UTC m=+70.530674646 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/137885a5-ec67-4cfd-baa1-2a43a9389bb1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z65fp" (UID: "137885a5-ec67-4cfd-baa1-2a43a9389bb1") : configmap "kube-rbac-proxy" not found
Jul 28 17:10:20.256219 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:10:20.256093    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/137885a5-ec67-4cfd-baa1-2a43a9389bb1-insights-runtime-extractor-tls podName:137885a5-ec67-4cfd-baa1-2a43a9389bb1 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:22.256084969 +0000 UTC m=+70.530701378 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/137885a5-ec67-4cfd-baa1-2a43a9389bb1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z65fp" (UID: "137885a5-ec67-4cfd-baa1-2a43a9389bb1") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:10:20.585190 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:20.585098    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zkf8x" event={"ID":"9a7b47e4-993e-4527-a00f-4ebbbc050348","Type":"ContainerStarted","Data":"76b6ae9aac22e7daaf50e132db219050f32cc7dd907c6edad9c799aff3f3545e"}
Jul 28 17:10:20.585190 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:20.585160    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zkf8x" event={"ID":"9a7b47e4-993e-4527-a00f-4ebbbc050348","Type":"ContainerStarted","Data":"9066e43ccb12926232fe63f346ae8af0e657d81d7d6dbe7dfcef47a5808717b3"}
Jul 28 17:10:20.620553 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:20.620526    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:20.719705 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:20.719673    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:20.808468 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:20.808436    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:21.183168 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:21.183128    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:21.362522 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:21.362478    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:21.515435 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:21.515355    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:22.274696 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:22.274653    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/137885a5-ec67-4cfd-baa1-2a43a9389bb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z65fp\" (UID: \"137885a5-ec67-4cfd-baa1-2a43a9389bb1\") " pod="openshift-insights/insights-runtime-extractor-z65fp"
Jul 28 17:10:22.275188 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:22.274703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/137885a5-ec67-4cfd-baa1-2a43a9389bb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z65fp\" (UID: \"137885a5-ec67-4cfd-baa1-2a43a9389bb1\") " pod="openshift-insights/insights-runtime-extractor-z65fp"
Jul 28 17:10:22.275382 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:22.275361    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/137885a5-ec67-4cfd-baa1-2a43a9389bb1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z65fp\" (UID: \"137885a5-ec67-4cfd-baa1-2a43a9389bb1\") " pod="openshift-insights/insights-runtime-extractor-z65fp"
Jul 28 17:10:22.277434 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:22.277412    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/137885a5-ec67-4cfd-baa1-2a43a9389bb1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z65fp\" (UID: \"137885a5-ec67-4cfd-baa1-2a43a9389bb1\") " pod="openshift-insights/insights-runtime-extractor-z65fp"
Jul 28 17:10:22.315350 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:22.315323    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mvdj9_81dc569e-3e52-4bb3-8f79-d58b183bb100/dns/0.log"
Jul 28 17:10:22.482410 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:22.482366    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z65fp"
Jul 28 17:10:22.516992 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:22.516960    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mvdj9_81dc569e-3e52-4bb3-8f79-d58b183bb100/kube-rbac-proxy/0.log"
Jul 28 17:10:22.715890 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:22.715861    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dbnhf_601b853c-8800-428b-89f6-1cb7b378600c/dns-node-resolver/0.log"
Jul 28 17:10:23.253770 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:23.253701    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zkf8x" podStartSLOduration=69.836859187 podStartE2EDuration="1m11.253679248s" totalImagesPullingTime="1.416820061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:20.604606261 +0000 UTC m=+68.879222692" watchObservedRunningTime="2026-07-28 17:10:23.253679248 +0000 UTC m=+71.528295685"
Jul 28 17:10:23.254830 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:23.254804    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z65fp"]
Jul 28 17:10:23.257984 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:10:23.257956    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137885a5_ec67_4cfd_baa1_2a43a9389bb1.slice/crio-3909ab35598723ac8f21bbb4fe05ae20942e3078322e155302e28e84c70fcba5 WatchSource:0}: Error finding container 3909ab35598723ac8f21bbb4fe05ae20942e3078322e155302e28e84c70fcba5: Status 404 returned error can't find the container with id 3909ab35598723ac8f21bbb4fe05ae20942e3078322e155302e28e84c70fcba5
Jul 28 17:10:23.287428 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:23.287400    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:23.597211 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:23.597179    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jwmpm" event={"ID":"189dbf08-647d-42a8-975f-39838ad4edf9","Type":"ContainerStarted","Data":"e71d23698ad19502811c54bfdbf502c44859e11a374c6fdf6ac75df5679dea11"}
Jul 28 17:10:23.598464 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:23.598433    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z65fp" event={"ID":"137885a5-ec67-4cfd-baa1-2a43a9389bb1","Type":"ContainerStarted","Data":"3909ab35598723ac8f21bbb4fe05ae20942e3078322e155302e28e84c70fcba5"}
Jul 28 17:10:23.599767 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:23.599748    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2nt29" event={"ID":"33068506-3820-4d64-bc6b-33ea180dca70","Type":"ContainerStarted","Data":"99cf48adc7aded98b290ea618f61f4564f74bcb57ea14e7a0bec151195ba7020"}
Jul 28 17:10:23.599878 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:23.599865    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:10:23.628717 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:23.628678    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2nt29" podStartSLOduration=67.101104546 podStartE2EDuration="1m11.628664158s" totalImagesPullingTime="4.527559612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:23.627681643 +0000 UTC m=+71.902298075" watchObservedRunningTime="2026-07-28 17:10:23.628664158 +0000 UTC m=+71.903280591"
Jul 28 17:10:23.628812 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:23.628772    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jwmpm" podStartSLOduration=67.093665978 podStartE2EDuration="1m11.628767458s" totalImagesPullingTime="4.53510148s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:23.61262444 +0000 UTC m=+71.887240865" watchObservedRunningTime="2026-07-28 17:10:23.628767458 +0000 UTC m=+71.903383889"
Jul 28 17:10:23.716093 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:23.716069    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ffvhq_6d4cfcb5-839a-431f-946c-6b5ac4e6c61e/node-ca/0.log"
Jul 28 17:10:24.317540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:24.317513    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5555ff8d6c-6mzsx_496b869a-13ac-4171-8068-7c839c75b1e9/router/0.log"
Jul 28 17:10:24.916916 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:24.916883    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h6pjb_baf5cf61-87a1-442f-981a-8d24b1197ec8/serve-healthcheck-canary/0.log"
Jul 28 17:10:25.689069 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:25.689036    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:25.941358 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:25.941280    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:26.091297 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.091260    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:26.182204 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.182166    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:26.233757 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.233684    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qw4bp"]
Jul 28 17:10:26.249457 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.249421    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.253401 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.253372    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 17:10:26.253621 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.253597    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bvlvp\""
Jul 28 17:10:26.253794 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.253771    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 17:10:26.255027 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.255005    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 17:10:26.255247 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.255233    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 17:10:26.255411 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.255398    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 17:10:26.255621 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.255603    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 17:10:26.269547 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.268403    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:26.306150 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.306125    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34af819b-df81-4984-81ca-d05c55376b97-sys\") pod \"node-exporter-qw4bp\" (UID: \"34af819b-df81-4984-81ca-d05c55376b97\") " pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.306311 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.306181    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/34af819b-df81-4984-81ca-d05c55376b97-root\") pod \"node-exporter-qw4bp\" (UID: \"34af819b-df81-4984-81ca-d05c55376b97\") " pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.306311 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.306212    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/34af819b-df81-4984-81ca-d05c55376b97-node-exporter-tls\") pod \"node-exporter-qw4bp\" (UID: \"34af819b-df81-4984-81ca-d05c55376b97\") " pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.306311 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.306240    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34af819b-df81-4984-81ca-d05c55376b97-metrics-client-ca\") pod \"node-exporter-qw4bp\" (UID: \"34af819b-df81-4984-81ca-d05c55376b97\") " pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.306311 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.306273    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34af819b-df81-4984-81ca-d05c55376b97-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qw4bp\" (UID: \"34af819b-df81-4984-81ca-d05c55376b97\") " pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.306311 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.306297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjkxc\" (UniqueName: \"kubernetes.io/projected/34af819b-df81-4984-81ca-d05c55376b97-kube-api-access-pjkxc\") pod \"node-exporter-qw4bp\" (UID: \"34af819b-df81-4984-81ca-d05c55376b97\") " pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.306562 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.306321    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/34af819b-df81-4984-81ca-d05c55376b97-node-exporter-wtmp\") pod \"node-exporter-qw4bp\" (UID: \"34af819b-df81-4984-81ca-d05c55376b97\") " pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.306562 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.306349    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/34af819b-df81-4984-81ca-d05c55376b97-node-exporter-textfile\") pod \"node-exporter-qw4bp\" (UID: \"34af819b-df81-4984-81ca-d05c55376b97\") " pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.306562 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.306378    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/34af819b-df81-4984-81ca-d05c55376b97-node-exporter-accelerators-collector-config\") pod \"node-exporter-qw4bp\" (UID: \"34af819b-df81-4984-81ca-d05c55376b97\") " pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.361099 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.361067    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:26.408964 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.407327    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/34af819b-df81-4984-81ca-d05c55376b97-root\") pod \"node-exporter-qw4bp\" (UID: \"34af819b-df81-4984-81ca-d05c55376b97\") " pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.408964 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.407379    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/34af819b-df81-4984-81ca-d05c55376b97-node-exporter-tls\") pod \"node-exporter-qw4bp\" (UID: \"34af819b-df81-4984-81ca-d05c55376b97\") " pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.408964 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.407402    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34af819b-df81-4984-81ca-d05c55376b97-metrics-client-ca\") pod \"node-exporter-qw4bp\" (UID: \"34af819b-df81-4984-81ca-d05c55376b97\") " pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.408964 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.407428    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34af819b-df81-4984-81ca-d05c55376b97-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qw4bp\" (UID: \"34af819b-df81-4984-81ca-d05c55376b97\") " pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.408964 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.407450    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjkxc\" (UniqueName: \"kubernetes.io/projected/34af819b-df81-4984-81ca-d05c55376b97-kube-api-access-pjkxc\") pod \"node-exporter-qw4bp\" (UID: \"34af819b-df81-4984-81ca-d05c55376b97\") " pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.408964 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.407477    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/34af819b-df81-4984-81ca-d05c55376b97-node-exporter-wtmp\") pod \"node-exporter-qw4bp\" (UID: \"34af819b-df81-4984-81ca-d05c55376b97\") " pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.408964 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.407507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/34af819b-df81-4984-81ca-d05c55376b97-node-exporter-textfile\") pod \"node-exporter-qw4bp\" (UID: \"34af819b-df81-4984-81ca-d05c55376b97\") " pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.408964 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.407536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/34af819b-df81-4984-81ca-d05c55376b97-node-exporter-accelerators-collector-config\") pod \"node-exporter-qw4bp\" (UID: \"34af819b-df81-4984-81ca-d05c55376b97\") " pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.408964 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.407561    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34af819b-df81-4984-81ca-d05c55376b97-sys\") pod \"node-exporter-qw4bp\" (UID: \"34af819b-df81-4984-81ca-d05c55376b97\") " pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.408964 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.407622    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34af819b-df81-4984-81ca-d05c55376b97-sys\") pod \"node-exporter-qw4bp\" (UID: \"34af819b-df81-4984-81ca-d05c55376b97\") " pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.408964 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.407662    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/34af819b-df81-4984-81ca-d05c55376b97-root\") pod \"node-exporter-qw4bp\" (UID: \"34af819b-df81-4984-81ca-d05c55376b97\") " pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.408964 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:10:26.407755    2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 17:10:26.408964 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:10:26.407803    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34af819b-df81-4984-81ca-d05c55376b97-node-exporter-tls podName:34af819b-df81-4984-81ca-d05c55376b97 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:26.907788036 +0000 UTC m=+75.182404445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/34af819b-df81-4984-81ca-d05c55376b97-node-exporter-tls") pod "node-exporter-qw4bp" (UID: "34af819b-df81-4984-81ca-d05c55376b97") : secret "node-exporter-tls" not found
Jul 28 17:10:26.408964 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.408083    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/34af819b-df81-4984-81ca-d05c55376b97-node-exporter-wtmp\") pod \"node-exporter-qw4bp\" (UID: \"34af819b-df81-4984-81ca-d05c55376b97\") " pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.408964 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.408308    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/34af819b-df81-4984-81ca-d05c55376b97-node-exporter-textfile\") pod \"node-exporter-qw4bp\" (UID: \"34af819b-df81-4984-81ca-d05c55376b97\") " pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.408964 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.408588    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34af819b-df81-4984-81ca-d05c55376b97-metrics-client-ca\") pod \"node-exporter-qw4bp\" (UID: \"34af819b-df81-4984-81ca-d05c55376b97\") " pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.408964 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.408737    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/34af819b-df81-4984-81ca-d05c55376b97-node-exporter-accelerators-collector-config\") pod \"node-exporter-qw4bp\" (UID: \"34af819b-df81-4984-81ca-d05c55376b97\") " pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.412309 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.412266    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34af819b-df81-4984-81ca-d05c55376b97-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qw4bp\" (UID: \"34af819b-df81-4984-81ca-d05c55376b97\") " pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.429014 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.428462    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjkxc\" (UniqueName: \"kubernetes.io/projected/34af819b-df81-4984-81ca-d05c55376b97-kube-api-access-pjkxc\") pod \"node-exporter-qw4bp\" (UID: \"34af819b-df81-4984-81ca-d05c55376b97\") " pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.461484 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.461419    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:26.617972 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.617870    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:26.713972 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.713920    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:26.911569 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.911529    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/34af819b-df81-4984-81ca-d05c55376b97-node-exporter-tls\") pod \"node-exporter-qw4bp\" (UID: \"34af819b-df81-4984-81ca-d05c55376b97\") " pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.914123 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.914096    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/34af819b-df81-4984-81ca-d05c55376b97-node-exporter-tls\") pod \"node-exporter-qw4bp\" (UID: \"34af819b-df81-4984-81ca-d05c55376b97\") " pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:26.917145 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:26.917117    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:27.163261 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:27.163182    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qw4bp"
Jul 28 17:10:27.185409 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:27.185375    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:27.296816 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:27.296784    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:27.498916 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:27.498845    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:27.598306 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:27.598269    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:28.507605 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:10:28.507575    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34af819b_df81_4984_81ca_d05c55376b97.slice/crio-e3a7aa7b1912f840b174355eea37e33a2d487653c7d92c715f55b0ba704c4fe0 WatchSource:0}: Error finding container e3a7aa7b1912f840b174355eea37e33a2d487653c7d92c715f55b0ba704c4fe0: Status 404 returned error can't find the container with id e3a7aa7b1912f840b174355eea37e33a2d487653c7d92c715f55b0ba704c4fe0
Jul 28 17:10:28.622263 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:28.622237    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z65fp" event={"ID":"137885a5-ec67-4cfd-baa1-2a43a9389bb1","Type":"ContainerStarted","Data":"033f0db5fa8ac93e50806d36dede1fc5b28ea73689bf2562a05495b3e2ad69b3"}
Jul 28 17:10:28.623536 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:28.623497    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qw4bp" event={"ID":"34af819b-df81-4984-81ca-d05c55376b97","Type":"ContainerStarted","Data":"e3a7aa7b1912f840b174355eea37e33a2d487653c7d92c715f55b0ba704c4fe0"}
Jul 28 17:10:29.212769 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:29.212668    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:29.311853 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:29.311764    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:29.638356 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:29.638327    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:29.740059 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:29.740027    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:29.840182 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:29.840153    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:29.943430 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:29.943396    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:30.057873 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:30.057835    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:30.146177 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:30.146147    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:30.633210 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:30.633176    2581 generic.go:356] "Generic (PLEG): container finished" podID="34af819b-df81-4984-81ca-d05c55376b97" containerID="09bb32d17ed9c251385608862706828cd83105a45635f4645e63cc0bdbd0110d" exitCode=0
Jul 28 17:10:30.633389 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:30.633254    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qw4bp" event={"ID":"34af819b-df81-4984-81ca-d05c55376b97","Type":"ContainerDied","Data":"09bb32d17ed9c251385608862706828cd83105a45635f4645e63cc0bdbd0110d"}
Jul 28 17:10:30.969345 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:30.969307    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:31.085248 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:31.085220    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:31.171379 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:31.171346    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:31.272906 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:31.272823    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:31.354133 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:31.354085    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:31.884147 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:31.884112    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:31.946475 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:31.946379    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:32.447802 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:32.447775    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:32.552306 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:32.552274    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:32.644252 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:32.644217    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qw4bp" event={"ID":"34af819b-df81-4984-81ca-d05c55376b97","Type":"ContainerStarted","Data":"09b68df538d4346e2a2e4a734bb2936e3abaf0d711394bea8b336a26beb6f5eb"}
Jul 28 17:10:32.644436 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:32.644312    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qw4bp" event={"ID":"34af819b-df81-4984-81ca-d05c55376b97","Type":"ContainerStarted","Data":"83fcb9b4792ed28d1b69aeffd922256e9a19e66a093f537d8186178cdba0a852"}
Jul 28 17:10:32.646188 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:32.646163    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z65fp" event={"ID":"137885a5-ec67-4cfd-baa1-2a43a9389bb1","Type":"ContainerStarted","Data":"4d015f9ec575818d032d8f05b96fa1d8a50b01a5e2c6324eb0b509f92d3808ae"}
Jul 28 17:10:32.669145 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:32.669093    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qw4bp" podStartSLOduration=5.479727579 podStartE2EDuration="6.66907968s" totalImagesPullingTime="1.115717002s" totalInitContainerRuntime="73.635099ms" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:32.66770801 +0000 UTC m=+80.942324442" watchObservedRunningTime="2026-07-28 17:10:32.66907968 +0000 UTC m=+80.943696111"
Jul 28 17:10:32.727214 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:32.727135    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:32.800636 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:32.800603    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:33.305156 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:33.305113    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:34.742804 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:34.742762    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:34.844545 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:34.844484    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:36.661992 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:36.661961    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z65fp" event={"ID":"137885a5-ec67-4cfd-baa1-2a43a9389bb1","Type":"ContainerStarted","Data":"6d1ae533e9f4335c115814d6de442876eb1310e2fe37e947874d16da141db5fc"}
Jul 28 17:10:36.681764 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:36.681712    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-z65fp" podStartSLOduration=6.4412126 podStartE2EDuration="18.681694966s" totalImagesPullingTime="12.240482366s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:36.680847867 +0000 UTC m=+84.955464299" watchObservedRunningTime="2026-07-28 17:10:36.681694966 +0000 UTC m=+84.956311400"
Jul 28 17:10:40.374539 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:40.374506    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:40.530125 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:40.530089    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:40.934826 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:40.934783    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:40.998362 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:40.998320    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:41.186631 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:41.186551    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:41.287731 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:41.287682    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:43.718528 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:43.718494    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:43.783368 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:43.783323    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:46.292296 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:46.292254    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:46.393431 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:46.393375    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:51.288179 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:51.288142    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:51.388631 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:51.388595    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:10:54.605844 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:10:54.605813    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2nt29"
Jul 28 17:11:01.290155 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:11:01.290116    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:01.380618 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:11:01.380583    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:11.468494 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:11:11.468461    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:11.568966 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:11:11.568903    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:11.668592 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:11:11.668557    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:12.121979 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:11:12.121927    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:12.191351 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:11:12.191314    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:42.430916 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:11:42.430858    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:42.510775 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:11:42.510748    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:48.785062 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:11:48.785026    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:49.085976 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:11:49.085872    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:51.128617 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:11:51.128584    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:51.475993 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:11:51.475965    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:54.371250 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:11:54.371211    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:11:54.473288 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:11:54.473249    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:00.350129 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:12:00.350091    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:00.424522 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:12:00.424495    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:01.361057 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:12:01.361021    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:01.452456 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:12:01.452417    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:01.553621 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:12:01.553593    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:11.646027 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:12:11.645878    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:11.748038 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:12:11.747983    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:18.197102 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:12:18.197060    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:22.814012 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:12:22.813975    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:51.213893 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:12:51.213861    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:12:51.314852 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:12:51.314820    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:32.807351 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:13:32.807313    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:32.872929 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:13:32.872899    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:33.154379 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:13:33.154352    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:33.255537 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:13:33.255495    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:33.362114 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:13:33.362086    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:56.497428 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:13:56.497390    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:13:56.595764 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:13:56.595723    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:02.638640 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:14:02.638588    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:02.964284 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:14:02.964228    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:03.193346 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:14:03.193312    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:03.328835 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:14:03.328732    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:03.680524 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:14:03.680489    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:05.350799 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:14:05.350768    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:14:12.145955 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:14:12.145910    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 17:14:12.149357 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:14:12.149334    2581 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 17:14:12.157676 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:14:12.157651    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 17:14:44.067080 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:14:44.067048    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:23.221259 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:23.221168    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:23.440934 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:23.440897    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-f5q4d"]
Jul 28 17:15:23.443876 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:23.443858    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-f5q4d"
Jul 28 17:15:23.448795 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:23.448772    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 17:15:23.448923 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:23.448824    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:23.449054 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:23.449017    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-cl2bb\""
Jul 28 17:15:23.450406 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:23.450089    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 17:15:23.450406 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:23.450121    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 17:15:23.458204 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:23.458185    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-f5q4d"]
Jul 28 17:15:23.520866 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:23.520791    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:23.551266 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:23.551230    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e6604f28-7469-40d5-a52b-ddc3fb14156f-data\") pod \"seaweedfs-64568bcd49-f5q4d\" (UID: \"e6604f28-7469-40d5-a52b-ddc3fb14156f\") " pod="kserve/seaweedfs-64568bcd49-f5q4d"
Jul 28 17:15:23.551381 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:23.551316    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmm77\" (UniqueName: \"kubernetes.io/projected/e6604f28-7469-40d5-a52b-ddc3fb14156f-kube-api-access-fmm77\") pod \"seaweedfs-64568bcd49-f5q4d\" (UID: \"e6604f28-7469-40d5-a52b-ddc3fb14156f\") " pod="kserve/seaweedfs-64568bcd49-f5q4d"
Jul 28 17:15:23.621859 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:23.621831    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:23.652003 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:23.651979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmm77\" (UniqueName: \"kubernetes.io/projected/e6604f28-7469-40d5-a52b-ddc3fb14156f-kube-api-access-fmm77\") pod \"seaweedfs-64568bcd49-f5q4d\" (UID: \"e6604f28-7469-40d5-a52b-ddc3fb14156f\") " pod="kserve/seaweedfs-64568bcd49-f5q4d"
Jul 28 17:15:23.652087 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:23.652013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e6604f28-7469-40d5-a52b-ddc3fb14156f-data\") pod \"seaweedfs-64568bcd49-f5q4d\" (UID: \"e6604f28-7469-40d5-a52b-ddc3fb14156f\") " pod="kserve/seaweedfs-64568bcd49-f5q4d"
Jul 28 17:15:23.652384 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:23.652367    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e6604f28-7469-40d5-a52b-ddc3fb14156f-data\") pod \"seaweedfs-64568bcd49-f5q4d\" (UID: \"e6604f28-7469-40d5-a52b-ddc3fb14156f\") " pod="kserve/seaweedfs-64568bcd49-f5q4d"
Jul 28 17:15:23.671333 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:23.671308    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmm77\" (UniqueName: \"kubernetes.io/projected/e6604f28-7469-40d5-a52b-ddc3fb14156f-kube-api-access-fmm77\") pod \"seaweedfs-64568bcd49-f5q4d\" (UID: \"e6604f28-7469-40d5-a52b-ddc3fb14156f\") " pod="kserve/seaweedfs-64568bcd49-f5q4d"
Jul 28 17:15:23.752926 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:23.752884    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-f5q4d"
Jul 28 17:15:23.883014 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:23.882986    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-f5q4d"]
Jul 28 17:15:23.884384 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:15:23.884357    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6604f28_7469_40d5_a52b_ddc3fb14156f.slice/crio-540c8422aab9a62b301f5668a3d44db5ce7e4922a429747964493ab1a4b24ea8 WatchSource:0}: Error finding container 540c8422aab9a62b301f5668a3d44db5ce7e4922a429747964493ab1a4b24ea8: Status 404 returned error can't find the container with id 540c8422aab9a62b301f5668a3d44db5ce7e4922a429747964493ab1a4b24ea8
Jul 28 17:15:23.885746 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:23.885724    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:15:24.173994 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:24.173937    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:24.265407 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:24.265374    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:24.339634 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:24.339606    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:24.574080 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:24.573984    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-f5q4d" event={"ID":"e6604f28-7469-40d5-a52b-ddc3fb14156f","Type":"ContainerStarted","Data":"540c8422aab9a62b301f5668a3d44db5ce7e4922a429747964493ab1a4b24ea8"}
Jul 28 17:15:27.584159 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:27.584117    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-f5q4d" event={"ID":"e6604f28-7469-40d5-a52b-ddc3fb14156f","Type":"ContainerStarted","Data":"95a73c5695b6cae17aee2341f4e2bb6181095091044df3e6e680f56de97cb1cf"}
Jul 28 17:15:27.584538 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:27.584259    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-f5q4d"
Jul 28 17:15:27.611684 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:27.611597    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-f5q4d" podStartSLOduration=1.15541881 podStartE2EDuration="4.611583957s" totalImagesPullingTime="3.456165147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:15:27.611571033 +0000 UTC m=+375.886187465" watchObservedRunningTime="2026-07-28 17:15:27.611583957 +0000 UTC m=+375.886200425"
Jul 28 17:15:33.590291 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:33.590257    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-f5q4d"
Jul 28 17:15:48.119510 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:48.119471    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:48.173288 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:48.173258    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:48.246836 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:48.246802    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:48.853067 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:48.853029    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:48.953610 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:48.953582    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:59.515345 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:59.515313    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:59.573361 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:59.573328    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:15:59.625414 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:15:59.625384    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:21.888479 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:16:21.888445    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:41.544145 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:16:41.544025    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:41.579522 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:16:41.579493    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-86b8n"]
Jul 28 17:16:41.582682 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:16:41.582660    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-86b8n"
Jul 28 17:16:41.585799 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:16:41.585770    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 28 17:16:41.586002 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:16:41.585976    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-h8hgz\""
Jul 28 17:16:41.592973 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:16:41.592931    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-86b8n"]
Jul 28 17:16:41.619440 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:16:41.619420    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:41.672280 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:16:41.672250    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kx4t\" (UniqueName: \"kubernetes.io/projected/a1da0f27-245f-4a42-816c-30ddc49048f5-kube-api-access-5kx4t\") pod \"model-serving-api-7b954ffc95-86b8n\" (UID: \"a1da0f27-245f-4a42-816c-30ddc49048f5\") " pod="kserve/model-serving-api-7b954ffc95-86b8n"
Jul 28 17:16:41.672388 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:16:41.672306    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1da0f27-245f-4a42-816c-30ddc49048f5-tls-certs\") pod \"model-serving-api-7b954ffc95-86b8n\" (UID: \"a1da0f27-245f-4a42-816c-30ddc49048f5\") " pod="kserve/model-serving-api-7b954ffc95-86b8n"
Jul 28 17:16:41.675920 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:16:41.675897    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:41.773334 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:16:41.773305    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kx4t\" (UniqueName: \"kubernetes.io/projected/a1da0f27-245f-4a42-816c-30ddc49048f5-kube-api-access-5kx4t\") pod \"model-serving-api-7b954ffc95-86b8n\" (UID: \"a1da0f27-245f-4a42-816c-30ddc49048f5\") " pod="kserve/model-serving-api-7b954ffc95-86b8n"
Jul 28 17:16:41.773422 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:16:41.773362    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1da0f27-245f-4a42-816c-30ddc49048f5-tls-certs\") pod \"model-serving-api-7b954ffc95-86b8n\" (UID: \"a1da0f27-245f-4a42-816c-30ddc49048f5\") " pod="kserve/model-serving-api-7b954ffc95-86b8n"
Jul 28 17:16:41.773490 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:16:41.773477    2581 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 28 17:16:41.773549 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:16:41.773539    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1da0f27-245f-4a42-816c-30ddc49048f5-tls-certs podName:a1da0f27-245f-4a42-816c-30ddc49048f5 nodeName:}" failed. No retries permitted until 2026-07-28 17:16:42.273520937 +0000 UTC m=+450.548137351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a1da0f27-245f-4a42-816c-30ddc49048f5-tls-certs") pod "model-serving-api-7b954ffc95-86b8n" (UID: "a1da0f27-245f-4a42-816c-30ddc49048f5") : secret "model-serving-api-tls" not found
Jul 28 17:16:41.785303 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:16:41.785272    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kx4t\" (UniqueName: \"kubernetes.io/projected/a1da0f27-245f-4a42-816c-30ddc49048f5-kube-api-access-5kx4t\") pod \"model-serving-api-7b954ffc95-86b8n\" (UID: \"a1da0f27-245f-4a42-816c-30ddc49048f5\") " pod="kserve/model-serving-api-7b954ffc95-86b8n"
Jul 28 17:16:42.192023 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:16:42.191991    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:42.251694 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:16:42.251669    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:16:42.276555 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:16:42.276532    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1da0f27-245f-4a42-816c-30ddc49048f5-tls-certs\") pod \"model-serving-api-7b954ffc95-86b8n\" (UID: \"a1da0f27-245f-4a42-816c-30ddc49048f5\") " pod="kserve/model-serving-api-7b954ffc95-86b8n"
Jul 28 17:16:42.278848 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:16:42.278824    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1da0f27-245f-4a42-816c-30ddc49048f5-tls-certs\") pod \"model-serving-api-7b954ffc95-86b8n\" (UID: \"a1da0f27-245f-4a42-816c-30ddc49048f5\") " pod="kserve/model-serving-api-7b954ffc95-86b8n"
Jul 28 17:16:42.492864 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:16:42.492783    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-86b8n"
Jul 28 17:16:42.615418 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:16:42.615386    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-86b8n"]
Jul 28 17:16:42.618727 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:16:42.618690    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1da0f27_245f_4a42_816c_30ddc49048f5.slice/crio-f5132803642746be5fd6635810c51b75d9bddb352f62c1a3029507f4f4f02db3 WatchSource:0}: Error finding container f5132803642746be5fd6635810c51b75d9bddb352f62c1a3029507f4f4f02db3: Status 404 returned error can't find the container with id f5132803642746be5fd6635810c51b75d9bddb352f62c1a3029507f4f4f02db3
Jul 28 17:16:42.826607 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:16:42.826518    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-86b8n" event={"ID":"a1da0f27-245f-4a42-816c-30ddc49048f5","Type":"ContainerStarted","Data":"f5132803642746be5fd6635810c51b75d9bddb352f62c1a3029507f4f4f02db3"}
Jul 28 17:16:45.838358 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:16:45.838321    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-86b8n" event={"ID":"a1da0f27-245f-4a42-816c-30ddc49048f5","Type":"ContainerStarted","Data":"95fc0cb530a54500744ae2de27f6bd50366886003db329178a59d6c3f1f526a2"}
Jul 28 17:16:45.838816 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:16:45.838422    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-86b8n"
Jul 28 17:16:45.855234 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:16:45.855148    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-86b8n" podStartSLOduration=2.548096005 podStartE2EDuration="4.85513279s" totalImagesPullingTime="2.307036785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:16:45.854837146 +0000 UTC m=+454.129453577" watchObservedRunningTime="2026-07-28 17:16:45.85513279 +0000 UTC m=+454.129749222"
Jul 28 17:16:56.846394 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:16:56.846360    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-86b8n"
Jul 28 17:17:11.300762 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:17:11.300724    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:28.806231 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:18:28.806130    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:34.291817 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:18:34.291781    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:44.806731 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:18:44.806697    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:44.877707 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:18:44.877680    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:57.668714 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:18:57.668679    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-pldmn"]
Jul 28 17:18:57.672673 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:18:57.672651    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-pldmn"
Jul 28 17:18:57.675359 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:18:57.675336    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Jul 28 17:18:57.675466 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:18:57.675339    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 28 17:18:57.678980 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:18:57.678649    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:57.682213 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:18:57.682192    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-pldmn"]
Jul 28 17:18:57.745025 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:18:57.744984    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:18:57.786196 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:18:57.786173    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/9e9cf802-1539-4f3a-a439-6894d2ccff0c-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-pldmn\" (UID: \"9e9cf802-1539-4f3a-a439-6894d2ccff0c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-pldmn"
Jul 28 17:18:57.786267 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:18:57.786213    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9e9cf802-1539-4f3a-a439-6894d2ccff0c-data\") pod \"seaweedfs-tls-serving-67b7477c6d-pldmn\" (UID: \"9e9cf802-1539-4f3a-a439-6894d2ccff0c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-pldmn"
Jul 28 17:18:57.786267 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:18:57.786237    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phkqs\" (UniqueName: \"kubernetes.io/projected/9e9cf802-1539-4f3a-a439-6894d2ccff0c-kube-api-access-phkqs\") pod \"seaweedfs-tls-serving-67b7477c6d-pldmn\" (UID: \"9e9cf802-1539-4f3a-a439-6894d2ccff0c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-pldmn"
Jul 28 17:18:57.887427 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:18:57.887396    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/9e9cf802-1539-4f3a-a439-6894d2ccff0c-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-pldmn\" (UID: \"9e9cf802-1539-4f3a-a439-6894d2ccff0c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-pldmn"
Jul 28 17:18:57.887540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:18:57.887439    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9e9cf802-1539-4f3a-a439-6894d2ccff0c-data\") pod \"seaweedfs-tls-serving-67b7477c6d-pldmn\" (UID: \"9e9cf802-1539-4f3a-a439-6894d2ccff0c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-pldmn"
Jul 28 17:18:57.887540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:18:57.887457    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phkqs\" (UniqueName: \"kubernetes.io/projected/9e9cf802-1539-4f3a-a439-6894d2ccff0c-kube-api-access-phkqs\") pod \"seaweedfs-tls-serving-67b7477c6d-pldmn\" (UID: \"9e9cf802-1539-4f3a-a439-6894d2ccff0c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-pldmn"
Jul 28 17:18:57.887849 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:18:57.887829    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9e9cf802-1539-4f3a-a439-6894d2ccff0c-data\") pod \"seaweedfs-tls-serving-67b7477c6d-pldmn\" (UID: \"9e9cf802-1539-4f3a-a439-6894d2ccff0c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-pldmn"
Jul 28 17:18:57.889769 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:18:57.889749    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/9e9cf802-1539-4f3a-a439-6894d2ccff0c-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-pldmn\" (UID: \"9e9cf802-1539-4f3a-a439-6894d2ccff0c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-pldmn"
Jul 28 17:18:57.896220 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:18:57.896201    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phkqs\" (UniqueName: \"kubernetes.io/projected/9e9cf802-1539-4f3a-a439-6894d2ccff0c-kube-api-access-phkqs\") pod \"seaweedfs-tls-serving-67b7477c6d-pldmn\" (UID: \"9e9cf802-1539-4f3a-a439-6894d2ccff0c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-pldmn"
Jul 28 17:18:57.981851 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:18:57.981788    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-pldmn"
Jul 28 17:18:58.105134 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:18:58.105102    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-pldmn"]
Jul 28 17:18:58.108437 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:18:58.108404    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e9cf802_1539_4f3a_a439_6894d2ccff0c.slice/crio-4fd6c883d4fb0bb08bab85dc4320f4e35c61c5b5333b90e8898e40ccab409f9d WatchSource:0}: Error finding container 4fd6c883d4fb0bb08bab85dc4320f4e35c61c5b5333b90e8898e40ccab409f9d: Status 404 returned error can't find the container with id 4fd6c883d4fb0bb08bab85dc4320f4e35c61c5b5333b90e8898e40ccab409f9d
Jul 28 17:18:58.268079 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:18:58.268011    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-pldmn" event={"ID":"9e9cf802-1539-4f3a-a439-6894d2ccff0c","Type":"ContainerStarted","Data":"4fd6c883d4fb0bb08bab85dc4320f4e35c61c5b5333b90e8898e40ccab409f9d"}
Jul 28 17:18:59.272457 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:18:59.272424    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-pldmn" event={"ID":"9e9cf802-1539-4f3a-a439-6894d2ccff0c","Type":"ContainerStarted","Data":"0cc5a9e2ef775bc560f0a78011110d81ccf28932501dfe0ae1dfc3d5ac381e71"}
Jul 28 17:18:59.294986 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:18:59.294921    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-pldmn" podStartSLOduration=2.027273974 podStartE2EDuration="2.294908125s" totalImagesPullingTime="267.634151ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:59.293212674 +0000 UTC m=+587.567829105" watchObservedRunningTime="2026-07-28 17:18:59.294908125 +0000 UTC m=+587.569524557"
Jul 28 17:19:12.187128 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:12.187088    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 17:19:12.194099 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:12.194073    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 17:19:15.508750 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:15.508715    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:16.736276 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:16.736220    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:27.052794 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.052757    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:27.179633 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.179596    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"]
Jul 28 17:19:27.183026 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.183006    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"
Jul 28 17:19:27.186720 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.186695    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:19:27.191638 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.191599    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:19:27.191754 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.191619    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:19:27.191850 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.191829    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:19:27.192147 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.192129    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 28 17:19:27.192259 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.192227    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 28 17:19:27.192968 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.192934    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 17:19:27.196073 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.196051    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"]
Jul 28 17:19:27.275545 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.275518    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:19:27.291802 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.291780    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd884657-0e2c-433e-90ca-faf810323f0a-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-5rkj2\" (UID: \"dd884657-0e2c-433e-90ca-faf810323f0a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"
Jul 28 17:19:27.291875 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.291822    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8pnt\" (UniqueName: \"kubernetes.io/projected/dd884657-0e2c-433e-90ca-faf810323f0a-kube-api-access-d8pnt\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-5rkj2\" (UID: \"dd884657-0e2c-433e-90ca-faf810323f0a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"
Jul 28 17:19:27.291926 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.291908    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd884657-0e2c-433e-90ca-faf810323f0a-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-5rkj2\" (UID: \"dd884657-0e2c-433e-90ca-faf810323f0a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"
Jul 28 17:19:27.291992 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.291936    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd884657-0e2c-433e-90ca-faf810323f0a-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-5rkj2\" (UID: \"dd884657-0e2c-433e-90ca-faf810323f0a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"
Jul 28 17:19:27.291992 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.291983    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd884657-0e2c-433e-90ca-faf810323f0a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-5rkj2\" (UID: \"dd884657-0e2c-433e-90ca-faf810323f0a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"
Jul 28 17:19:27.392550 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.392486    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd884657-0e2c-433e-90ca-faf810323f0a-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-5rkj2\" (UID: \"dd884657-0e2c-433e-90ca-faf810323f0a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"
Jul 28 17:19:27.392550 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.392517    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd884657-0e2c-433e-90ca-faf810323f0a-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-5rkj2\" (UID: \"dd884657-0e2c-433e-90ca-faf810323f0a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"
Jul 28 17:19:27.392550 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.392539    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd884657-0e2c-433e-90ca-faf810323f0a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-5rkj2\" (UID: \"dd884657-0e2c-433e-90ca-faf810323f0a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"
Jul 28 17:19:27.392774 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.392573    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd884657-0e2c-433e-90ca-faf810323f0a-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-5rkj2\" (UID: \"dd884657-0e2c-433e-90ca-faf810323f0a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"
Jul 28 17:19:27.392774 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.392610    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8pnt\" (UniqueName: \"kubernetes.io/projected/dd884657-0e2c-433e-90ca-faf810323f0a-kube-api-access-d8pnt\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-5rkj2\" (UID: \"dd884657-0e2c-433e-90ca-faf810323f0a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"
Jul 28 17:19:27.392933 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.392911    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd884657-0e2c-433e-90ca-faf810323f0a-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-5rkj2\" (UID: \"dd884657-0e2c-433e-90ca-faf810323f0a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"
Jul 28 17:19:27.393218 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.393198    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd884657-0e2c-433e-90ca-faf810323f0a-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-5rkj2\" (UID: \"dd884657-0e2c-433e-90ca-faf810323f0a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"
Jul 28 17:19:27.393294 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.393246    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd884657-0e2c-433e-90ca-faf810323f0a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-5rkj2\" (UID: \"dd884657-0e2c-433e-90ca-faf810323f0a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"
Jul 28 17:19:27.394878 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.394860    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd884657-0e2c-433e-90ca-faf810323f0a-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-5rkj2\" (UID: \"dd884657-0e2c-433e-90ca-faf810323f0a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"
Jul 28 17:19:27.402833 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.402804    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8pnt\" (UniqueName: \"kubernetes.io/projected/dd884657-0e2c-433e-90ca-faf810323f0a-kube-api-access-d8pnt\") pod \"isvc-sklearn-batcher-predictor-8b78f8788-5rkj2\" (UID: \"dd884657-0e2c-433e-90ca-faf810323f0a\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"
Jul 28 17:19:27.497030 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.497006    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"
Jul 28 17:19:27.625566 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:27.625535    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"]
Jul 28 17:19:27.629355 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:19:27.629327    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd884657_0e2c_433e_90ca_faf810323f0a.slice/crio-ee119f0c62f1d977c7e64ecd534f21692d5caf4192e6a5dd669c778125dc62c8 WatchSource:0}: Error finding container ee119f0c62f1d977c7e64ecd534f21692d5caf4192e6a5dd669c778125dc62c8: Status 404 returned error can't find the container with id ee119f0c62f1d977c7e64ecd534f21692d5caf4192e6a5dd669c778125dc62c8
Jul 28 17:19:28.368078 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:28.368040    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" event={"ID":"dd884657-0e2c-433e-90ca-faf810323f0a","Type":"ContainerStarted","Data":"ee119f0c62f1d977c7e64ecd534f21692d5caf4192e6a5dd669c778125dc62c8"}
Jul 28 17:19:31.382335 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:31.382255    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" event={"ID":"dd884657-0e2c-433e-90ca-faf810323f0a","Type":"ContainerStarted","Data":"5498fa2c122417b8cec4be07781a2811625f586263280297bf7a25cc695eb940"}
Jul 28 17:19:35.397686 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:35.397653    2581 generic.go:356] "Generic (PLEG): container finished" podID="dd884657-0e2c-433e-90ca-faf810323f0a" containerID="5498fa2c122417b8cec4be07781a2811625f586263280297bf7a25cc695eb940" exitCode=0
Jul 28 17:19:35.398075 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:35.397711    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" event={"ID":"dd884657-0e2c-433e-90ca-faf810323f0a","Type":"ContainerDied","Data":"5498fa2c122417b8cec4be07781a2811625f586263280297bf7a25cc695eb940"}
Jul 28 17:19:49.459398 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:49.459357    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" event={"ID":"dd884657-0e2c-433e-90ca-faf810323f0a","Type":"ContainerStarted","Data":"2d22729f3fee321cd1b560511c845524c136cbdf3d69b3e2ca733333c6b40ae8"}
Jul 28 17:19:52.472590 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:52.472556    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" event={"ID":"dd884657-0e2c-433e-90ca-faf810323f0a","Type":"ContainerStarted","Data":"6e0d13e4ce669ff4bfef3945b1d4ce4c354045817799c221d4283affdcfa5c30"}
Jul 28 17:19:55.487918 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:55.487877    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" event={"ID":"dd884657-0e2c-433e-90ca-faf810323f0a","Type":"ContainerStarted","Data":"a7c0d01e0e54ce14c5434f1c772f6322f4691f6893b760735b4332556a977e11"}
Jul 28 17:19:55.488334 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:55.488117    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"
Jul 28 17:19:55.488334 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:55.488195    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"
Jul 28 17:19:55.489553 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:55.489520    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 28 17:19:55.513714 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:55.513670    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podStartSLOduration=2.569713775 podStartE2EDuration="28.513656703s" totalImagesPullingTime="22.358961978s" totalInitContainerRuntime="3.58498095s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:19:27 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:19:55.511974052 +0000 UTC m=+643.786590483" watchObservedRunningTime="2026-07-28 17:19:55.513656703 +0000 UTC m=+643.788273135"
Jul 28 17:19:56.492011 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:56.491974    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"
Jul 28 17:19:56.492488 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:56.492081    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 28 17:19:56.492985 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:56.492935    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:19:56.495959 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:56.495920    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"
Jul 28 17:19:57.495324 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:57.495282    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 28 17:19:57.495714 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:57.495608    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:19:58.499130 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:58.499092    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 28 17:19:58.499564 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:19:58.499311    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:20:08.499432 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:20:08.499378    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 28 17:20:08.499928 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:20:08.499806    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:20:18.499156 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:20:18.499106    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 28 17:20:18.499545 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:20:18.499482    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:20:28.499438 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:20:28.499391    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 28 17:20:28.499895 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:20:28.499799    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:20:38.499808 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:20:38.499758    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 28 17:20:38.500244 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:20:38.500169    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:20:48.499132 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:20:48.499085    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 28 17:20:48.499624 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:20:48.499524    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:20:51.799341 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:20:51.799298    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:20:58.499036 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:20:58.498985    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 28 17:20:58.499523 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:20:58.499353    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:21:07.286144 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:07.286066    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"
Jul 28 17:21:07.286508 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:07.286163    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"
Jul 28 17:21:22.272591 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.272556    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:22.339960 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.339913    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"]
Jul 28 17:21:22.340252 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.340223    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="kserve-container" containerID="cri-o://2d22729f3fee321cd1b560511c845524c136cbdf3d69b3e2ca733333c6b40ae8" gracePeriod=30
Jul 28 17:21:22.340252 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.340223    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="kube-rbac-proxy" containerID="cri-o://6e0d13e4ce669ff4bfef3945b1d4ce4c354045817799c221d4283affdcfa5c30" gracePeriod=30
Jul 28 17:21:22.340394 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.340296    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="agent" containerID="cri-o://a7c0d01e0e54ce14c5434f1c772f6322f4691f6893b760735b4332556a977e11" gracePeriod=30
Jul 28 17:21:22.358701 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.358667    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:22.369848 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.369822    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"]
Jul 28 17:21:22.373525 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.373500    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"
Jul 28 17:21:22.376353 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.376327    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 28 17:21:22.376353 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.376339    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 28 17:21:22.388470 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.388447    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"]
Jul 28 17:21:22.425174 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.425136    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ae10ed1-044f-4843-b9db-af7e7a9860bf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9\" (UID: \"2ae10ed1-044f-4843-b9db-af7e7a9860bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"
Jul 28 17:21:22.425351 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.425177    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ae10ed1-044f-4843-b9db-af7e7a9860bf-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9\" (UID: \"2ae10ed1-044f-4843-b9db-af7e7a9860bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"
Jul 28 17:21:22.425351 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.425216    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ae10ed1-044f-4843-b9db-af7e7a9860bf-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9\" (UID: \"2ae10ed1-044f-4843-b9db-af7e7a9860bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"
Jul 28 17:21:22.425351 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.425258    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ae10ed1-044f-4843-b9db-af7e7a9860bf-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9\" (UID: \"2ae10ed1-044f-4843-b9db-af7e7a9860bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"
Jul 28 17:21:22.425351 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.425326    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwwql\" (UniqueName: \"kubernetes.io/projected/2ae10ed1-044f-4843-b9db-af7e7a9860bf-kube-api-access-qwwql\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9\" (UID: \"2ae10ed1-044f-4843-b9db-af7e7a9860bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"
Jul 28 17:21:22.514222 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.514193    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:21:22.526132 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.526063    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwwql\" (UniqueName: \"kubernetes.io/projected/2ae10ed1-044f-4843-b9db-af7e7a9860bf-kube-api-access-qwwql\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9\" (UID: \"2ae10ed1-044f-4843-b9db-af7e7a9860bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"
Jul 28 17:21:22.526132 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.526102    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ae10ed1-044f-4843-b9db-af7e7a9860bf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9\" (UID: \"2ae10ed1-044f-4843-b9db-af7e7a9860bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"
Jul 28 17:21:22.526132 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.526120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ae10ed1-044f-4843-b9db-af7e7a9860bf-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9\" (UID: \"2ae10ed1-044f-4843-b9db-af7e7a9860bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"
Jul 28 17:21:22.526349 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.526147    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ae10ed1-044f-4843-b9db-af7e7a9860bf-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9\" (UID: \"2ae10ed1-044f-4843-b9db-af7e7a9860bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"
Jul 28 17:21:22.526349 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.526183    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ae10ed1-044f-4843-b9db-af7e7a9860bf-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9\" (UID: \"2ae10ed1-044f-4843-b9db-af7e7a9860bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"
Jul 28 17:21:22.526705 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.526684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ae10ed1-044f-4843-b9db-af7e7a9860bf-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9\" (UID: \"2ae10ed1-044f-4843-b9db-af7e7a9860bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"
Jul 28 17:21:22.526782 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.526740    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ae10ed1-044f-4843-b9db-af7e7a9860bf-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9\" (UID: \"2ae10ed1-044f-4843-b9db-af7e7a9860bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"
Jul 28 17:21:22.526782 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.526759    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ae10ed1-044f-4843-b9db-af7e7a9860bf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9\" (UID: \"2ae10ed1-044f-4843-b9db-af7e7a9860bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"
Jul 28 17:21:22.528558 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.528542    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ae10ed1-044f-4843-b9db-af7e7a9860bf-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9\" (UID: \"2ae10ed1-044f-4843-b9db-af7e7a9860bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"
Jul 28 17:21:22.535767 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.535744    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwwql\" (UniqueName: \"kubernetes.io/projected/2ae10ed1-044f-4843-b9db-af7e7a9860bf-kube-api-access-qwwql\") pod \"isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9\" (UID: \"2ae10ed1-044f-4843-b9db-af7e7a9860bf\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"
Jul 28 17:21:22.683080 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.683043    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"
Jul 28 17:21:22.789064 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.788996    2581 generic.go:356] "Generic (PLEG): container finished" podID="dd884657-0e2c-433e-90ca-faf810323f0a" containerID="6e0d13e4ce669ff4bfef3945b1d4ce4c354045817799c221d4283affdcfa5c30" exitCode=2
Jul 28 17:21:22.789179 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.789064    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" event={"ID":"dd884657-0e2c-433e-90ca-faf810323f0a","Type":"ContainerDied","Data":"6e0d13e4ce669ff4bfef3945b1d4ce4c354045817799c221d4283affdcfa5c30"}
Jul 28 17:21:22.816274 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:22.816250    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"]
Jul 28 17:21:23.793896 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:23.793860    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" event={"ID":"2ae10ed1-044f-4843-b9db-af7e7a9860bf","Type":"ContainerStarted","Data":"a13b40052b101915d8071e05b0554f3297dbe0bb9ecbaa27000331856013e530"}
Jul 28 17:21:23.794269 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:23.793934    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" event={"ID":"2ae10ed1-044f-4843-b9db-af7e7a9860bf","Type":"ContainerStarted","Data":"3028358f0b4c4dc2bb347a972e684437c4b3a1838895828238ffe2cdf5a5de5a"}
Jul 28 17:21:26.492961 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:26.492898    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.17:8643/healthz\": dial tcp 10.133.0.17:8643: connect: connection refused"
Jul 28 17:21:26.805265 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:26.805235    2581 generic.go:356] "Generic (PLEG): container finished" podID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerID="a13b40052b101915d8071e05b0554f3297dbe0bb9ecbaa27000331856013e530" exitCode=0
Jul 28 17:21:26.805379 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:26.805308    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" event={"ID":"2ae10ed1-044f-4843-b9db-af7e7a9860bf","Type":"ContainerDied","Data":"a13b40052b101915d8071e05b0554f3297dbe0bb9ecbaa27000331856013e530"}
Jul 28 17:21:26.807868 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:26.807845    2581 generic.go:356] "Generic (PLEG): container finished" podID="dd884657-0e2c-433e-90ca-faf810323f0a" containerID="2d22729f3fee321cd1b560511c845524c136cbdf3d69b3e2ca733333c6b40ae8" exitCode=0
Jul 28 17:21:26.807936 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:26.807901    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" event={"ID":"dd884657-0e2c-433e-90ca-faf810323f0a","Type":"ContainerDied","Data":"2d22729f3fee321cd1b560511c845524c136cbdf3d69b3e2ca733333c6b40ae8"}
Jul 28 17:21:27.285908 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:27.285864    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 28 17:21:27.286148 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:27.286123    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:21:27.815493 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:27.815462    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" event={"ID":"2ae10ed1-044f-4843-b9db-af7e7a9860bf","Type":"ContainerStarted","Data":"7fd52bf78498c474df78c126ea1f8d0704a79fa7f0636a306621f72177d54c57"}
Jul 28 17:21:27.815865 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:27.815550    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" event={"ID":"2ae10ed1-044f-4843-b9db-af7e7a9860bf","Type":"ContainerStarted","Data":"e15283e4d4d12b3188d16581b1e197d210c5f2ba9df0a36a3b35021441460928"}
Jul 28 17:21:27.815865 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:27.815574    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" event={"ID":"2ae10ed1-044f-4843-b9db-af7e7a9860bf","Type":"ContainerStarted","Data":"283259ca6eab42016e6113bc131a249f6dc60db0ad3f7340349288e510550811"}
Jul 28 17:21:27.815966 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:27.815865    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"
Jul 28 17:21:27.816028 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:27.816009    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"
Jul 28 17:21:27.817221 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:27.817192    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:5000: connect: connection refused"
Jul 28 17:21:27.842227 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:27.842180    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podStartSLOduration=1.941872042 podStartE2EDuration="5.842168088s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.900296046s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:21:27.840116562 +0000 UTC m=+736.114732993" watchObservedRunningTime="2026-07-28 17:21:27.842168088 +0000 UTC m=+736.116784519"
Jul 28 17:21:28.820203 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:28.820173    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"
Jul 28 17:21:28.820592 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:28.820297    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:5000: connect: connection refused"
Jul 28 17:21:28.821256 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:28.821237    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:21:29.823683 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:29.823646    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:5000: connect: connection refused"
Jul 28 17:21:29.824115 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:29.824063    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:21:31.492631 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:31.492591    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.17:8643/healthz\": dial tcp 10.133.0.17:8643: connect: connection refused"
Jul 28 17:21:34.833244 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:34.833214    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"
Jul 28 17:21:34.833878 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:34.833846    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:5000: connect: connection refused"
Jul 28 17:21:34.834222 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:34.834201    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:21:36.492080 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:36.492031    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.17:8643/healthz\": dial tcp 10.133.0.17:8643: connect: connection refused"
Jul 28 17:21:36.492460 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:36.492163    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"
Jul 28 17:21:37.285640 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:37.285599    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 28 17:21:37.285909 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:37.285886    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:21:41.492406 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:41.492364    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.17:8643/healthz\": dial tcp 10.133.0.17:8643: connect: connection refused"
Jul 28 17:21:44.834436 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:44.834387    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:5000: connect: connection refused"
Jul 28 17:21:44.834879 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:44.834851    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:21:46.492358 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:46.492313    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.17:8643/healthz\": dial tcp 10.133.0.17:8643: connect: connection refused"
Jul 28 17:21:47.286492 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:47.286448    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.17:8080: connect: connection refused"
Jul 28 17:21:47.286685 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:47.286578    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"
Jul 28 17:21:47.286815 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:47.286798    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:21:47.286877 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:47.286867    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"
Jul 28 17:21:51.492852 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:51.492806    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.17:8643/healthz\": dial tcp 10.133.0.17:8643: connect: connection refused"
Jul 28 17:21:52.484771 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.484743    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"
Jul 28 17:21:52.536854 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.536823    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8pnt\" (UniqueName: \"kubernetes.io/projected/dd884657-0e2c-433e-90ca-faf810323f0a-kube-api-access-d8pnt\") pod \"dd884657-0e2c-433e-90ca-faf810323f0a\" (UID: \"dd884657-0e2c-433e-90ca-faf810323f0a\") "
Jul 28 17:21:52.537302 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.536861    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd884657-0e2c-433e-90ca-faf810323f0a-kube-rbac-proxy-sar-config\") pod \"dd884657-0e2c-433e-90ca-faf810323f0a\" (UID: \"dd884657-0e2c-433e-90ca-faf810323f0a\") "
Jul 28 17:21:52.537302 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.536891    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd884657-0e2c-433e-90ca-faf810323f0a-cabundle-cert\") pod \"dd884657-0e2c-433e-90ca-faf810323f0a\" (UID: \"dd884657-0e2c-433e-90ca-faf810323f0a\") "
Jul 28 17:21:52.537302 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.536979    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd884657-0e2c-433e-90ca-faf810323f0a-kserve-provision-location\") pod \"dd884657-0e2c-433e-90ca-faf810323f0a\" (UID: \"dd884657-0e2c-433e-90ca-faf810323f0a\") "
Jul 28 17:21:52.537302 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.537029    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd884657-0e2c-433e-90ca-faf810323f0a-proxy-tls\") pod \"dd884657-0e2c-433e-90ca-faf810323f0a\" (UID: \"dd884657-0e2c-433e-90ca-faf810323f0a\") "
Jul 28 17:21:52.537302 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.537276    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd884657-0e2c-433e-90ca-faf810323f0a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dd884657-0e2c-433e-90ca-faf810323f0a" (UID: "dd884657-0e2c-433e-90ca-faf810323f0a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:21:52.537499 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.537314    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd884657-0e2c-433e-90ca-faf810323f0a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dd884657-0e2c-433e-90ca-faf810323f0a" (UID: "dd884657-0e2c-433e-90ca-faf810323f0a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:21:52.537499 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.537319    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd884657-0e2c-433e-90ca-faf810323f0a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dd884657-0e2c-433e-90ca-faf810323f0a" (UID: "dd884657-0e2c-433e-90ca-faf810323f0a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:21:52.539015 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.538993    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd884657-0e2c-433e-90ca-faf810323f0a-kube-api-access-d8pnt" (OuterVolumeSpecName: "kube-api-access-d8pnt") pod "dd884657-0e2c-433e-90ca-faf810323f0a" (UID: "dd884657-0e2c-433e-90ca-faf810323f0a"). InnerVolumeSpecName "kube-api-access-d8pnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:21:52.539088 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.539018    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd884657-0e2c-433e-90ca-faf810323f0a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dd884657-0e2c-433e-90ca-faf810323f0a" (UID: "dd884657-0e2c-433e-90ca-faf810323f0a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:21:52.638034 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.637923    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8pnt\" (UniqueName: \"kubernetes.io/projected/dd884657-0e2c-433e-90ca-faf810323f0a-kube-api-access-d8pnt\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:21:52.638034 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.637985    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd884657-0e2c-433e-90ca-faf810323f0a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:21:52.638034 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.637996    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd884657-0e2c-433e-90ca-faf810323f0a-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:21:52.638034 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.638004    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd884657-0e2c-433e-90ca-faf810323f0a-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:21:52.638034 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.638013    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd884657-0e2c-433e-90ca-faf810323f0a-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:21:52.906134 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.906103    2581 generic.go:356] "Generic (PLEG): container finished" podID="dd884657-0e2c-433e-90ca-faf810323f0a" containerID="a7c0d01e0e54ce14c5434f1c772f6322f4691f6893b760735b4332556a977e11" exitCode=0
Jul 28 17:21:52.906313 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.906144    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" event={"ID":"dd884657-0e2c-433e-90ca-faf810323f0a","Type":"ContainerDied","Data":"a7c0d01e0e54ce14c5434f1c772f6322f4691f6893b760735b4332556a977e11"}
Jul 28 17:21:52.906313 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.906162    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"
Jul 28 17:21:52.906313 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.906215    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2" event={"ID":"dd884657-0e2c-433e-90ca-faf810323f0a","Type":"ContainerDied","Data":"ee119f0c62f1d977c7e64ecd534f21692d5caf4192e6a5dd669c778125dc62c8"}
Jul 28 17:21:52.906313 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.906255    2581 scope.go:117] "RemoveContainer" containerID="a7c0d01e0e54ce14c5434f1c772f6322f4691f6893b760735b4332556a977e11"
Jul 28 17:21:52.915227 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.915212    2581 scope.go:117] "RemoveContainer" containerID="6e0d13e4ce669ff4bfef3945b1d4ce4c354045817799c221d4283affdcfa5c30"
Jul 28 17:21:52.922249 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.922233    2581 scope.go:117] "RemoveContainer" containerID="2d22729f3fee321cd1b560511c845524c136cbdf3d69b3e2ca733333c6b40ae8"
Jul 28 17:21:52.928727 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.928710    2581 scope.go:117] "RemoveContainer" containerID="5498fa2c122417b8cec4be07781a2811625f586263280297bf7a25cc695eb940"
Jul 28 17:21:52.935206 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.935178    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"]
Jul 28 17:21:52.935555 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.935540    2581 scope.go:117] "RemoveContainer" containerID="a7c0d01e0e54ce14c5434f1c772f6322f4691f6893b760735b4332556a977e11"
Jul 28 17:21:52.935784 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:21:52.935767    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7c0d01e0e54ce14c5434f1c772f6322f4691f6893b760735b4332556a977e11\": container with ID starting with a7c0d01e0e54ce14c5434f1c772f6322f4691f6893b760735b4332556a977e11 not found: ID does not exist" containerID="a7c0d01e0e54ce14c5434f1c772f6322f4691f6893b760735b4332556a977e11"
Jul 28 17:21:52.935836 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.935791    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7c0d01e0e54ce14c5434f1c772f6322f4691f6893b760735b4332556a977e11"} err="failed to get container status \"a7c0d01e0e54ce14c5434f1c772f6322f4691f6893b760735b4332556a977e11\": rpc error: code = NotFound desc = could not find container \"a7c0d01e0e54ce14c5434f1c772f6322f4691f6893b760735b4332556a977e11\": container with ID starting with a7c0d01e0e54ce14c5434f1c772f6322f4691f6893b760735b4332556a977e11 not found: ID does not exist"
Jul 28 17:21:52.935836 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.935828    2581 scope.go:117] "RemoveContainer" containerID="6e0d13e4ce669ff4bfef3945b1d4ce4c354045817799c221d4283affdcfa5c30"
Jul 28 17:21:52.936073 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:21:52.936055    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e0d13e4ce669ff4bfef3945b1d4ce4c354045817799c221d4283affdcfa5c30\": container with ID starting with 6e0d13e4ce669ff4bfef3945b1d4ce4c354045817799c221d4283affdcfa5c30 not found: ID does not exist" containerID="6e0d13e4ce669ff4bfef3945b1d4ce4c354045817799c221d4283affdcfa5c30"
Jul 28 17:21:52.936123 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.936077    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e0d13e4ce669ff4bfef3945b1d4ce4c354045817799c221d4283affdcfa5c30"} err="failed to get container status \"6e0d13e4ce669ff4bfef3945b1d4ce4c354045817799c221d4283affdcfa5c30\": rpc error: code = NotFound desc = could not find container \"6e0d13e4ce669ff4bfef3945b1d4ce4c354045817799c221d4283affdcfa5c30\": container with ID starting with 6e0d13e4ce669ff4bfef3945b1d4ce4c354045817799c221d4283affdcfa5c30 not found: ID does not exist"
Jul 28 17:21:52.936123 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.936092    2581 scope.go:117] "RemoveContainer" containerID="2d22729f3fee321cd1b560511c845524c136cbdf3d69b3e2ca733333c6b40ae8"
Jul 28 17:21:52.936329 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:21:52.936309    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d22729f3fee321cd1b560511c845524c136cbdf3d69b3e2ca733333c6b40ae8\": container with ID starting with 2d22729f3fee321cd1b560511c845524c136cbdf3d69b3e2ca733333c6b40ae8 not found: ID does not exist" containerID="2d22729f3fee321cd1b560511c845524c136cbdf3d69b3e2ca733333c6b40ae8"
Jul 28 17:21:52.936378 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.936345    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d22729f3fee321cd1b560511c845524c136cbdf3d69b3e2ca733333c6b40ae8"} err="failed to get container status \"2d22729f3fee321cd1b560511c845524c136cbdf3d69b3e2ca733333c6b40ae8\": rpc error: code = NotFound desc = could not find container \"2d22729f3fee321cd1b560511c845524c136cbdf3d69b3e2ca733333c6b40ae8\": container with ID starting with 2d22729f3fee321cd1b560511c845524c136cbdf3d69b3e2ca733333c6b40ae8 not found: ID does not exist"
Jul 28 17:21:52.936378 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.936362    2581 scope.go:117] "RemoveContainer" containerID="5498fa2c122417b8cec4be07781a2811625f586263280297bf7a25cc695eb940"
Jul 28 17:21:52.936606 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:21:52.936578    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5498fa2c122417b8cec4be07781a2811625f586263280297bf7a25cc695eb940\": container with ID starting with 5498fa2c122417b8cec4be07781a2811625f586263280297bf7a25cc695eb940 not found: ID does not exist" containerID="5498fa2c122417b8cec4be07781a2811625f586263280297bf7a25cc695eb940"
Jul 28 17:21:52.936692 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.936601    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5498fa2c122417b8cec4be07781a2811625f586263280297bf7a25cc695eb940"} err="failed to get container status \"5498fa2c122417b8cec4be07781a2811625f586263280297bf7a25cc695eb940\": rpc error: code = NotFound desc = could not find container \"5498fa2c122417b8cec4be07781a2811625f586263280297bf7a25cc695eb940\": container with ID starting with 5498fa2c122417b8cec4be07781a2811625f586263280297bf7a25cc695eb940 not found: ID does not exist"
Jul 28 17:21:52.942218 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:52.942197    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8b78f8788-5rkj2"]
Jul 28 17:21:54.289347 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:54.289314    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" path="/var/lib/kubelet/pods/dd884657-0e2c-433e-90ca-faf810323f0a/volumes"
Jul 28 17:21:54.833841 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:54.833791    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:5000: connect: connection refused"
Jul 28 17:21:54.834237 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:21:54.834212    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:22:04.834015 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:04.833966    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:5000: connect: connection refused"
Jul 28 17:22:04.834411 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:04.834394    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:22:14.572460 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:14.572424    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:14.834023 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:14.833916    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:5000: connect: connection refused"
Jul 28 17:22:14.834370 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:14.834340    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:22:24.834328 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:24.834284    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:5000: connect: connection refused"
Jul 28 17:22:24.834805 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:24.834780    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:22:34.834159 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:34.834118    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:5000: connect: connection refused"
Jul 28 17:22:34.834621 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:34.834477    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:22:38.290095 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:38.290069    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"
Jul 28 17:22:38.290449 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:38.290148    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"
Jul 28 17:22:47.610673 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.610630    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:47.620891 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.620856    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj"]
Jul 28 17:22:47.621314 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.621293    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="kube-rbac-proxy"
Jul 28 17:22:47.621314 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.621314    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="kube-rbac-proxy"
Jul 28 17:22:47.621462 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.621328    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="kserve-container"
Jul 28 17:22:47.621462 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.621337    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="kserve-container"
Jul 28 17:22:47.621462 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.621365    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="storage-initializer"
Jul 28 17:22:47.621462 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.621374    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="storage-initializer"
Jul 28 17:22:47.621462 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.621390    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="agent"
Jul 28 17:22:47.621462 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.621398    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="agent"
Jul 28 17:22:47.621462 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.621453    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="agent"
Jul 28 17:22:47.621462 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.621463    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="kube-rbac-proxy"
Jul 28 17:22:47.621855 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.621478    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd884657-0e2c-433e-90ca-faf810323f0a" containerName="kserve-container"
Jul 28 17:22:47.624751 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.624730    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj"
Jul 28 17:22:47.627564 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.627542    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 28 17:22:47.627723 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.627703    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 28 17:22:47.633892 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.633866    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"]
Jul 28 17:22:47.634144 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.634123    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="kserve-container" containerID="cri-o://283259ca6eab42016e6113bc131a249f6dc60db0ad3f7340349288e510550811" gracePeriod=30
Jul 28 17:22:47.634220 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.634195    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="agent" containerID="cri-o://7fd52bf78498c474df78c126ea1f8d0704a79fa7f0636a306621f72177d54c57" gracePeriod=30
Jul 28 17:22:47.634288 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.634200    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="kube-rbac-proxy" containerID="cri-o://e15283e4d4d12b3188d16581b1e197d210c5f2ba9df0a36a3b35021441460928" gracePeriod=30
Jul 28 17:22:47.637359 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.637332    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj"]
Jul 28 17:22:47.687109 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.687073    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:47.735552 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.735520    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4966\" (UniqueName: \"kubernetes.io/projected/978898e3-c721-458e-8940-7f9b3a28d05b-kube-api-access-t4966\") pod \"message-dumper-predictor-6575bcfdfd-f9dbj\" (UID: \"978898e3-c721-458e-8940-7f9b3a28d05b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj"
Jul 28 17:22:47.735707 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.735565    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/978898e3-c721-458e-8940-7f9b3a28d05b-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-f9dbj\" (UID: \"978898e3-c721-458e-8940-7f9b3a28d05b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj"
Jul 28 17:22:47.735707 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.735626    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/978898e3-c721-458e-8940-7f9b3a28d05b-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-f9dbj\" (UID: \"978898e3-c721-458e-8940-7f9b3a28d05b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj"
Jul 28 17:22:47.836320 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.836287    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/978898e3-c721-458e-8940-7f9b3a28d05b-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-f9dbj\" (UID: \"978898e3-c721-458e-8940-7f9b3a28d05b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj"
Jul 28 17:22:47.836491 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.836354    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4966\" (UniqueName: \"kubernetes.io/projected/978898e3-c721-458e-8940-7f9b3a28d05b-kube-api-access-t4966\") pod \"message-dumper-predictor-6575bcfdfd-f9dbj\" (UID: \"978898e3-c721-458e-8940-7f9b3a28d05b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj"
Jul 28 17:22:47.836491 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.836383    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/978898e3-c721-458e-8940-7f9b3a28d05b-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-f9dbj\" (UID: \"978898e3-c721-458e-8940-7f9b3a28d05b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj"
Jul 28 17:22:47.836593 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:22:47.836499    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
Jul 28 17:22:47.836593 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:22:47.836570    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/978898e3-c721-458e-8940-7f9b3a28d05b-proxy-tls podName:978898e3-c721-458e-8940-7f9b3a28d05b nodeName:}" failed. No retries permitted until 2026-07-28 17:22:48.336538777 +0000 UTC m=+816.611155186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/978898e3-c721-458e-8940-7f9b3a28d05b-proxy-tls") pod "message-dumper-predictor-6575bcfdfd-f9dbj" (UID: "978898e3-c721-458e-8940-7f9b3a28d05b") : secret "message-dumper-predictor-serving-cert" not found
Jul 28 17:22:47.836990 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.836968    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/978898e3-c721-458e-8940-7f9b3a28d05b-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-f9dbj\" (UID: \"978898e3-c721-458e-8940-7f9b3a28d05b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj"
Jul 28 17:22:47.855767 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.855741    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4966\" (UniqueName: \"kubernetes.io/projected/978898e3-c721-458e-8940-7f9b3a28d05b-kube-api-access-t4966\") pod \"message-dumper-predictor-6575bcfdfd-f9dbj\" (UID: \"978898e3-c721-458e-8940-7f9b3a28d05b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj"
Jul 28 17:22:47.964886 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:47.964843    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:22:48.108233 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:48.108201    2581 generic.go:356] "Generic (PLEG): container finished" podID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerID="e15283e4d4d12b3188d16581b1e197d210c5f2ba9df0a36a3b35021441460928" exitCode=2
Jul 28 17:22:48.108422 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:48.108278    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" event={"ID":"2ae10ed1-044f-4843-b9db-af7e7a9860bf","Type":"ContainerDied","Data":"e15283e4d4d12b3188d16581b1e197d210c5f2ba9df0a36a3b35021441460928"}
Jul 28 17:22:48.290131 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:48.290032    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:5000: connect: connection refused"
Jul 28 17:22:48.290318 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:48.290285    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:22:48.341181 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:48.341150    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/978898e3-c721-458e-8940-7f9b3a28d05b-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-f9dbj\" (UID: \"978898e3-c721-458e-8940-7f9b3a28d05b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj"
Jul 28 17:22:48.343529 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:48.343509    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/978898e3-c721-458e-8940-7f9b3a28d05b-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-f9dbj\" (UID: \"978898e3-c721-458e-8940-7f9b3a28d05b\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj"
Jul 28 17:22:48.535325 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:48.535290    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj"
Jul 28 17:22:48.664126 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:48.664008    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj"]
Jul 28 17:22:48.666495 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:22:48.666467    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod978898e3_c721_458e_8940_7f9b3a28d05b.slice/crio-45611958eaeece522480cbce04023c34766431f59b7d2e0b3f6af89b4823766b WatchSource:0}: Error finding container 45611958eaeece522480cbce04023c34766431f59b7d2e0b3f6af89b4823766b: Status 404 returned error can't find the container with id 45611958eaeece522480cbce04023c34766431f59b7d2e0b3f6af89b4823766b
Jul 28 17:22:48.668406 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:48.668390    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:22:49.112319 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:49.112282    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj" event={"ID":"978898e3-c721-458e-8940-7f9b3a28d05b","Type":"ContainerStarted","Data":"45611958eaeece522480cbce04023c34766431f59b7d2e0b3f6af89b4823766b"}
Jul 28 17:22:49.824613 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:49.824576    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.18:8643/healthz\": dial tcp 10.133.0.18:8643: connect: connection refused"
Jul 28 17:22:50.117258 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:50.117229    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj" event={"ID":"978898e3-c721-458e-8940-7f9b3a28d05b","Type":"ContainerStarted","Data":"27e3abade736f9d5d4bad1877c2b1eab302bda08118d0ca4f295922ca8363231"}
Jul 28 17:22:50.117358 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:50.117293    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj" event={"ID":"978898e3-c721-458e-8940-7f9b3a28d05b","Type":"ContainerStarted","Data":"95d8a205fbff3d17d6b29298763e10ed96ddf7b80a75b36000276d5796b3ca84"}
Jul 28 17:22:50.117358 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:50.117329    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj"
Jul 28 17:22:50.148659 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:50.148619    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj" podStartSLOduration=1.834608875 podStartE2EDuration="3.14860426s" totalImagesPullingTime="1.313995385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:22:50.148232749 +0000 UTC m=+818.422849184" watchObservedRunningTime="2026-07-28 17:22:50.14860426 +0000 UTC m=+818.423220694"
Jul 28 17:22:51.120934 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:51.120899    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj"
Jul 28 17:22:51.122648 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:51.122626    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj"
Jul 28 17:22:52.125852 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:52.125821    2581 generic.go:356] "Generic (PLEG): container finished" podID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerID="283259ca6eab42016e6113bc131a249f6dc60db0ad3f7340349288e510550811" exitCode=0
Jul 28 17:22:52.126219 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:52.125894    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" event={"ID":"2ae10ed1-044f-4843-b9db-af7e7a9860bf","Type":"ContainerDied","Data":"283259ca6eab42016e6113bc131a249f6dc60db0ad3f7340349288e510550811"}
Jul 28 17:22:54.824270 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:54.824229    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.18:8643/healthz\": dial tcp 10.133.0.18:8643: connect: connection refused"
Jul 28 17:22:58.134589 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:58.134557    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj"
Jul 28 17:22:58.289890 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:58.289851    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:5000: connect: connection refused"
Jul 28 17:22:58.290215 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:58.290191    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:22:59.824537 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:59.824496    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.18:8643/healthz\": dial tcp 10.133.0.18:8643: connect: connection refused"
Jul 28 17:22:59.824982 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:22:59.824619    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"
Jul 28 17:23:04.824019 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:04.823979    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.18:8643/healthz\": dial tcp 10.133.0.18:8643: connect: connection refused"
Jul 28 17:23:07.533748 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:07.533713    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:07.580703 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:07.580663    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"]
Jul 28 17:23:07.587248 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:07.587229    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"
Jul 28 17:23:07.590190 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:07.590161    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 28 17:23:07.590294 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:07.590188    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 28 17:23:07.596787 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:07.596766    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"]
Jul 28 17:23:07.696473 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:07.696439    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:23:07.778631 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:07.778599    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48041153-9cc2-409a-b1a9-9cc566c8faec-cabundle-cert\") pod \"isvc-logger-predictor-866fbfd695-w6rrf\" (UID: \"48041153-9cc2-409a-b1a9-9cc566c8faec\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"
Jul 28 17:23:07.778762 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:07.778639    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48041153-9cc2-409a-b1a9-9cc566c8faec-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-866fbfd695-w6rrf\" (UID: \"48041153-9cc2-409a-b1a9-9cc566c8faec\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"
Jul 28 17:23:07.778762 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:07.778678    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2b8n\" (UniqueName: \"kubernetes.io/projected/48041153-9cc2-409a-b1a9-9cc566c8faec-kube-api-access-l2b8n\") pod \"isvc-logger-predictor-866fbfd695-w6rrf\" (UID: \"48041153-9cc2-409a-b1a9-9cc566c8faec\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"
Jul 28 17:23:07.778762 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:07.778699    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48041153-9cc2-409a-b1a9-9cc566c8faec-kserve-provision-location\") pod \"isvc-logger-predictor-866fbfd695-w6rrf\" (UID: \"48041153-9cc2-409a-b1a9-9cc566c8faec\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"
Jul 28 17:23:07.778762 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:07.778722    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48041153-9cc2-409a-b1a9-9cc566c8faec-proxy-tls\") pod \"isvc-logger-predictor-866fbfd695-w6rrf\" (UID: \"48041153-9cc2-409a-b1a9-9cc566c8faec\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"
Jul 28 17:23:07.879487 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:07.879417    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48041153-9cc2-409a-b1a9-9cc566c8faec-proxy-tls\") pod \"isvc-logger-predictor-866fbfd695-w6rrf\" (UID: \"48041153-9cc2-409a-b1a9-9cc566c8faec\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"
Jul 28 17:23:07.879487 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:07.879457    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48041153-9cc2-409a-b1a9-9cc566c8faec-cabundle-cert\") pod \"isvc-logger-predictor-866fbfd695-w6rrf\" (UID: \"48041153-9cc2-409a-b1a9-9cc566c8faec\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"
Jul 28 17:23:07.879487 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:07.879482    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48041153-9cc2-409a-b1a9-9cc566c8faec-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-866fbfd695-w6rrf\" (UID: \"48041153-9cc2-409a-b1a9-9cc566c8faec\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"
Jul 28 17:23:07.879693 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:07.879637    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2b8n\" (UniqueName: \"kubernetes.io/projected/48041153-9cc2-409a-b1a9-9cc566c8faec-kube-api-access-l2b8n\") pod \"isvc-logger-predictor-866fbfd695-w6rrf\" (UID: \"48041153-9cc2-409a-b1a9-9cc566c8faec\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"
Jul 28 17:23:07.879693 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:07.879678    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48041153-9cc2-409a-b1a9-9cc566c8faec-kserve-provision-location\") pod \"isvc-logger-predictor-866fbfd695-w6rrf\" (UID: \"48041153-9cc2-409a-b1a9-9cc566c8faec\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"
Jul 28 17:23:07.880045 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:07.880019    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48041153-9cc2-409a-b1a9-9cc566c8faec-kserve-provision-location\") pod \"isvc-logger-predictor-866fbfd695-w6rrf\" (UID: \"48041153-9cc2-409a-b1a9-9cc566c8faec\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"
Jul 28 17:23:07.880203 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:07.880185    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48041153-9cc2-409a-b1a9-9cc566c8faec-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-866fbfd695-w6rrf\" (UID: \"48041153-9cc2-409a-b1a9-9cc566c8faec\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"
Jul 28 17:23:07.880203 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:07.880193    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48041153-9cc2-409a-b1a9-9cc566c8faec-cabundle-cert\") pod \"isvc-logger-predictor-866fbfd695-w6rrf\" (UID: \"48041153-9cc2-409a-b1a9-9cc566c8faec\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"
Jul 28 17:23:07.881916 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:07.881899    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48041153-9cc2-409a-b1a9-9cc566c8faec-proxy-tls\") pod \"isvc-logger-predictor-866fbfd695-w6rrf\" (UID: \"48041153-9cc2-409a-b1a9-9cc566c8faec\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"
Jul 28 17:23:07.889087 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:07.889065    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2b8n\" (UniqueName: \"kubernetes.io/projected/48041153-9cc2-409a-b1a9-9cc566c8faec-kube-api-access-l2b8n\") pod \"isvc-logger-predictor-866fbfd695-w6rrf\" (UID: \"48041153-9cc2-409a-b1a9-9cc566c8faec\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"
Jul 28 17:23:07.897918 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:07.897897    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"
Jul 28 17:23:08.033642 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:08.033608    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"]
Jul 28 17:23:08.036318 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:23:08.036291    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48041153_9cc2_409a_b1a9_9cc566c8faec.slice/crio-17bc9befd7d0cefa67d908f9b4abaf75df8c66aa66a31332653447e47601b6ee WatchSource:0}: Error finding container 17bc9befd7d0cefa67d908f9b4abaf75df8c66aa66a31332653447e47601b6ee: Status 404 returned error can't find the container with id 17bc9befd7d0cefa67d908f9b4abaf75df8c66aa66a31332653447e47601b6ee
Jul 28 17:23:08.183135 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:08.183097    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" event={"ID":"48041153-9cc2-409a-b1a9-9cc566c8faec","Type":"ContainerStarted","Data":"844f25c5241885e6421fbab88b63c82d818ce559dd26bceb6bb60ce179ca5858"}
Jul 28 17:23:08.183272 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:08.183190    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" event={"ID":"48041153-9cc2-409a-b1a9-9cc566c8faec","Type":"ContainerStarted","Data":"17bc9befd7d0cefa67d908f9b4abaf75df8c66aa66a31332653447e47601b6ee"}
Jul 28 17:23:08.290636 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:08.290595    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:5000: connect: connection refused"
Jul 28 17:23:08.290806 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:08.290712    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"
Jul 28 17:23:08.291010 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:08.290981    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:23:08.291118 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:08.291080    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"
Jul 28 17:23:09.824459 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:09.824416    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.18:8643/healthz\": dial tcp 10.133.0.18:8643: connect: connection refused"
Jul 28 17:23:12.203541 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:12.203450    2581 generic.go:356] "Generic (PLEG): container finished" podID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerID="844f25c5241885e6421fbab88b63c82d818ce559dd26bceb6bb60ce179ca5858" exitCode=0
Jul 28 17:23:12.203541 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:12.203527    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" event={"ID":"48041153-9cc2-409a-b1a9-9cc566c8faec","Type":"ContainerDied","Data":"844f25c5241885e6421fbab88b63c82d818ce559dd26bceb6bb60ce179ca5858"}
Jul 28 17:23:13.209824 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:13.209783    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" event={"ID":"48041153-9cc2-409a-b1a9-9cc566c8faec","Type":"ContainerStarted","Data":"c58a59734988ace78c444b35d2576327bd8aacd134dc89846b2648c3837117c0"}
Jul 28 17:23:13.210207 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:13.209854    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" event={"ID":"48041153-9cc2-409a-b1a9-9cc566c8faec","Type":"ContainerStarted","Data":"b567d0c3c4c4e13a2ac0a54db0e3dba04fd69020d1a3206d9c02045287afc014"}
Jul 28 17:23:13.210207 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:13.209901    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" event={"ID":"48041153-9cc2-409a-b1a9-9cc566c8faec","Type":"ContainerStarted","Data":"07d3499eef83bf82b79dce485bcf0742a96e838a3342ff3a5819ce51a4cd6902"}
Jul 28 17:23:13.210303 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:13.210216    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"
Jul 28 17:23:13.210418 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:13.210402    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"
Jul 28 17:23:13.211532 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:13.211507    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 28 17:23:13.240248 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:13.240209    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podStartSLOduration=2.425847897 podStartE2EDuration="6.240196625s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.814348728s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:13.238408667 +0000 UTC m=+841.513025099" watchObservedRunningTime="2026-07-28 17:23:13.240196625 +0000 UTC m=+841.514813057"
Jul 28 17:23:14.214042 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:14.214007    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"
Jul 28 17:23:14.214423 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:14.214160    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 28 17:23:14.215313 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:14.215283    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:23:14.824736 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:14.824695    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.18:8643/healthz\": dial tcp 10.133.0.18:8643: connect: connection refused"
Jul 28 17:23:15.218084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:15.218046    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 28 17:23:15.218480 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:15.218438    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:23:17.787006 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:17.784459    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"
Jul 28 17:23:17.848409 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:17.848379    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ae10ed1-044f-4843-b9db-af7e7a9860bf-kserve-provision-location\") pod \"2ae10ed1-044f-4843-b9db-af7e7a9860bf\" (UID: \"2ae10ed1-044f-4843-b9db-af7e7a9860bf\") "
Jul 28 17:23:17.848561 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:17.848418    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ae10ed1-044f-4843-b9db-af7e7a9860bf-proxy-tls\") pod \"2ae10ed1-044f-4843-b9db-af7e7a9860bf\" (UID: \"2ae10ed1-044f-4843-b9db-af7e7a9860bf\") "
Jul 28 17:23:17.848561 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:17.848464    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwwql\" (UniqueName: \"kubernetes.io/projected/2ae10ed1-044f-4843-b9db-af7e7a9860bf-kube-api-access-qwwql\") pod \"2ae10ed1-044f-4843-b9db-af7e7a9860bf\" (UID: \"2ae10ed1-044f-4843-b9db-af7e7a9860bf\") "
Jul 28 17:23:17.848561 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:17.848499    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ae10ed1-044f-4843-b9db-af7e7a9860bf-kube-rbac-proxy-sar-config\") pod \"2ae10ed1-044f-4843-b9db-af7e7a9860bf\" (UID: \"2ae10ed1-044f-4843-b9db-af7e7a9860bf\") "
Jul 28 17:23:17.848561 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:17.848536    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ae10ed1-044f-4843-b9db-af7e7a9860bf-cabundle-cert\") pod \"2ae10ed1-044f-4843-b9db-af7e7a9860bf\" (UID: \"2ae10ed1-044f-4843-b9db-af7e7a9860bf\") "
Jul 28 17:23:17.848812 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:17.848784    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae10ed1-044f-4843-b9db-af7e7a9860bf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2ae10ed1-044f-4843-b9db-af7e7a9860bf" (UID: "2ae10ed1-044f-4843-b9db-af7e7a9860bf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:23:17.848935 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:17.848914    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae10ed1-044f-4843-b9db-af7e7a9860bf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2ae10ed1-044f-4843-b9db-af7e7a9860bf" (UID: "2ae10ed1-044f-4843-b9db-af7e7a9860bf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:23:17.849031 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:17.849009    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae10ed1-044f-4843-b9db-af7e7a9860bf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2ae10ed1-044f-4843-b9db-af7e7a9860bf" (UID: "2ae10ed1-044f-4843-b9db-af7e7a9860bf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:23:17.850534 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:17.850511    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae10ed1-044f-4843-b9db-af7e7a9860bf-kube-api-access-qwwql" (OuterVolumeSpecName: "kube-api-access-qwwql") pod "2ae10ed1-044f-4843-b9db-af7e7a9860bf" (UID: "2ae10ed1-044f-4843-b9db-af7e7a9860bf"). InnerVolumeSpecName "kube-api-access-qwwql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:23:17.850598 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:17.850531    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ae10ed1-044f-4843-b9db-af7e7a9860bf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2ae10ed1-044f-4843-b9db-af7e7a9860bf" (UID: "2ae10ed1-044f-4843-b9db-af7e7a9860bf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:23:17.949156 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:17.949112    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwwql\" (UniqueName: \"kubernetes.io/projected/2ae10ed1-044f-4843-b9db-af7e7a9860bf-kube-api-access-qwwql\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:23:17.949156 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:17.949152    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ae10ed1-044f-4843-b9db-af7e7a9860bf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:23:17.949361 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:17.949167    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ae10ed1-044f-4843-b9db-af7e7a9860bf-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:23:17.949361 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:17.949180    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ae10ed1-044f-4843-b9db-af7e7a9860bf-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:23:17.949361 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:17.949192    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ae10ed1-044f-4843-b9db-af7e7a9860bf-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:23:18.229972 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:18.229861    2581 generic.go:356] "Generic (PLEG): container finished" podID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerID="7fd52bf78498c474df78c126ea1f8d0704a79fa7f0636a306621f72177d54c57" exitCode=0
Jul 28 17:23:18.229972 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:18.229964    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"
Jul 28 17:23:18.230205 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:18.229964    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" event={"ID":"2ae10ed1-044f-4843-b9db-af7e7a9860bf","Type":"ContainerDied","Data":"7fd52bf78498c474df78c126ea1f8d0704a79fa7f0636a306621f72177d54c57"}
Jul 28 17:23:18.230205 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:18.230083    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9" event={"ID":"2ae10ed1-044f-4843-b9db-af7e7a9860bf","Type":"ContainerDied","Data":"3028358f0b4c4dc2bb347a972e684437c4b3a1838895828238ffe2cdf5a5de5a"}
Jul 28 17:23:18.230205 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:18.230130    2581 scope.go:117] "RemoveContainer" containerID="7fd52bf78498c474df78c126ea1f8d0704a79fa7f0636a306621f72177d54c57"
Jul 28 17:23:18.238766 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:18.238753    2581 scope.go:117] "RemoveContainer" containerID="e15283e4d4d12b3188d16581b1e197d210c5f2ba9df0a36a3b35021441460928"
Jul 28 17:23:18.245599 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:18.245583    2581 scope.go:117] "RemoveContainer" containerID="283259ca6eab42016e6113bc131a249f6dc60db0ad3f7340349288e510550811"
Jul 28 17:23:18.252246 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:18.252230    2581 scope.go:117] "RemoveContainer" containerID="a13b40052b101915d8071e05b0554f3297dbe0bb9ecbaa27000331856013e530"
Jul 28 17:23:18.255573 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:18.255553    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"]
Jul 28 17:23:18.258429 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:18.258411    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-849d7cc4c5-n69d9"]
Jul 28 17:23:18.260958 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:18.260915    2581 scope.go:117] "RemoveContainer" containerID="7fd52bf78498c474df78c126ea1f8d0704a79fa7f0636a306621f72177d54c57"
Jul 28 17:23:18.261212 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:23:18.261194    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fd52bf78498c474df78c126ea1f8d0704a79fa7f0636a306621f72177d54c57\": container with ID starting with 7fd52bf78498c474df78c126ea1f8d0704a79fa7f0636a306621f72177d54c57 not found: ID does not exist" containerID="7fd52bf78498c474df78c126ea1f8d0704a79fa7f0636a306621f72177d54c57"
Jul 28 17:23:18.261302 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:18.261218    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fd52bf78498c474df78c126ea1f8d0704a79fa7f0636a306621f72177d54c57"} err="failed to get container status \"7fd52bf78498c474df78c126ea1f8d0704a79fa7f0636a306621f72177d54c57\": rpc error: code = NotFound desc = could not find container \"7fd52bf78498c474df78c126ea1f8d0704a79fa7f0636a306621f72177d54c57\": container with ID starting with 7fd52bf78498c474df78c126ea1f8d0704a79fa7f0636a306621f72177d54c57 not found: ID does not exist"
Jul 28 17:23:18.261302 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:18.261235    2581 scope.go:117] "RemoveContainer" containerID="e15283e4d4d12b3188d16581b1e197d210c5f2ba9df0a36a3b35021441460928"
Jul 28 17:23:18.261476 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:23:18.261458    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e15283e4d4d12b3188d16581b1e197d210c5f2ba9df0a36a3b35021441460928\": container with ID starting with e15283e4d4d12b3188d16581b1e197d210c5f2ba9df0a36a3b35021441460928 not found: ID does not exist" containerID="e15283e4d4d12b3188d16581b1e197d210c5f2ba9df0a36a3b35021441460928"
Jul 28 17:23:18.261538 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:18.261478    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e15283e4d4d12b3188d16581b1e197d210c5f2ba9df0a36a3b35021441460928"} err="failed to get container status \"e15283e4d4d12b3188d16581b1e197d210c5f2ba9df0a36a3b35021441460928\": rpc error: code = NotFound desc = could not find container \"e15283e4d4d12b3188d16581b1e197d210c5f2ba9df0a36a3b35021441460928\": container with ID starting with e15283e4d4d12b3188d16581b1e197d210c5f2ba9df0a36a3b35021441460928 not found: ID does not exist"
Jul 28 17:23:18.261538 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:18.261493    2581 scope.go:117] "RemoveContainer" containerID="283259ca6eab42016e6113bc131a249f6dc60db0ad3f7340349288e510550811"
Jul 28 17:23:18.261721 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:23:18.261704    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"283259ca6eab42016e6113bc131a249f6dc60db0ad3f7340349288e510550811\": container with ID starting with 283259ca6eab42016e6113bc131a249f6dc60db0ad3f7340349288e510550811 not found: ID does not exist" containerID="283259ca6eab42016e6113bc131a249f6dc60db0ad3f7340349288e510550811"
Jul 28 17:23:18.261788 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:18.261724    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"283259ca6eab42016e6113bc131a249f6dc60db0ad3f7340349288e510550811"} err="failed to get container status \"283259ca6eab42016e6113bc131a249f6dc60db0ad3f7340349288e510550811\": rpc error: code = NotFound desc = could not find container \"283259ca6eab42016e6113bc131a249f6dc60db0ad3f7340349288e510550811\": container with ID starting with 283259ca6eab42016e6113bc131a249f6dc60db0ad3f7340349288e510550811 not found: ID does not exist"
Jul 28 17:23:18.261788 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:18.261736    2581 scope.go:117] "RemoveContainer" containerID="a13b40052b101915d8071e05b0554f3297dbe0bb9ecbaa27000331856013e530"
Jul 28 17:23:18.261929 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:23:18.261908    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a13b40052b101915d8071e05b0554f3297dbe0bb9ecbaa27000331856013e530\": container with ID starting with a13b40052b101915d8071e05b0554f3297dbe0bb9ecbaa27000331856013e530 not found: ID does not exist" containerID="a13b40052b101915d8071e05b0554f3297dbe0bb9ecbaa27000331856013e530"
Jul 28 17:23:18.262019 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:18.261931    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a13b40052b101915d8071e05b0554f3297dbe0bb9ecbaa27000331856013e530"} err="failed to get container status \"a13b40052b101915d8071e05b0554f3297dbe0bb9ecbaa27000331856013e530\": rpc error: code = NotFound desc = could not find container \"a13b40052b101915d8071e05b0554f3297dbe0bb9ecbaa27000331856013e530\": container with ID starting with a13b40052b101915d8071e05b0554f3297dbe0bb9ecbaa27000331856013e530 not found: ID does not exist"
Jul 28 17:23:18.289733 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:18.289709    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" path="/var/lib/kubelet/pods/2ae10ed1-044f-4843-b9db-af7e7a9860bf/volumes"
Jul 28 17:23:20.221977 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:20.221928    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"
Jul 28 17:23:20.222496 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:20.222466    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 28 17:23:20.222805 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:20.222780    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:23:30.222635 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:30.222590    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 28 17:23:30.223147 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:30.223074    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:23:40.222732 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:40.222679    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 28 17:23:40.223216 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:40.223134    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:23:50.222586 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:50.222538    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 28 17:23:50.223224 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:23:50.223202    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:24:00.222545 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:00.222500    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 28 17:24:00.222988 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:00.222967    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:24:10.222889 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:10.222800    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 28 17:24:10.223359 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:10.223294    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:24:12.213115 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:12.213085    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 17:24:12.214905 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:12.214885    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 17:24:20.222505 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:20.222464    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 28 17:24:20.223047 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:20.222903    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:24:30.223689 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:30.223659    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"
Jul 28 17:24:30.224146 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:30.223960    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"
Jul 28 17:24:42.605187 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:42.605156    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-f9dbj_978898e3-c721-458e-8940-7f9b3a28d05b/kserve-container/0.log"
Jul 28 17:24:42.863621 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:42.863550    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"]
Jul 28 17:24:42.863824 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:42.863793    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="kserve-container" containerID="cri-o://07d3499eef83bf82b79dce485bcf0742a96e838a3342ff3a5819ce51a4cd6902" gracePeriod=30
Jul 28 17:24:42.863969 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:42.863820    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="agent" containerID="cri-o://c58a59734988ace78c444b35d2576327bd8aacd134dc89846b2648c3837117c0" gracePeriod=30
Jul 28 17:24:42.863969 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:42.863837    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="kube-rbac-proxy" containerID="cri-o://b567d0c3c4c4e13a2ac0a54db0e3dba04fd69020d1a3206d9c02045287afc014" gracePeriod=30
Jul 28 17:24:42.959547 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:42.959513    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj"]
Jul 28 17:24:42.959793 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:42.959767    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj" podUID="978898e3-c721-458e-8940-7f9b3a28d05b" containerName="kserve-container" containerID="cri-o://95d8a205fbff3d17d6b29298763e10ed96ddf7b80a75b36000276d5796b3ca84" gracePeriod=30
Jul 28 17:24:42.959793 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:42.959772    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj" podUID="978898e3-c721-458e-8940-7f9b3a28d05b" containerName="kube-rbac-proxy" containerID="cri-o://27e3abade736f9d5d4bad1877c2b1eab302bda08118d0ca4f295922ca8363231" gracePeriod=30
Jul 28 17:24:43.027377 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.027349    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"]
Jul 28 17:24:43.027625 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.027613    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="kube-rbac-proxy"
Jul 28 17:24:43.027670 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.027626    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="kube-rbac-proxy"
Jul 28 17:24:43.027670 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.027643    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="storage-initializer"
Jul 28 17:24:43.027670 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.027649    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="storage-initializer"
Jul 28 17:24:43.027670 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.027659    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="kserve-container"
Jul 28 17:24:43.027670 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.027665    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="kserve-container"
Jul 28 17:24:43.027814 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.027673    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="agent"
Jul 28 17:24:43.027814 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.027678    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="agent"
Jul 28 17:24:43.027814 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.027719    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="agent"
Jul 28 17:24:43.027814 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.027727    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="kserve-container"
Jul 28 17:24:43.027814 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.027735    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ae10ed1-044f-4843-b9db-af7e7a9860bf" containerName="kube-rbac-proxy"
Jul 28 17:24:43.030590 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.030574    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"
Jul 28 17:24:43.034154 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.034133    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 28 17:24:43.034264 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.034194    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 28 17:24:43.047455 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.047432    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"]
Jul 28 17:24:43.066185 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.066160    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:43.073177 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.073157    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39c508a1-66a8-4b0e-960a-af901334fd55-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-m7tdn\" (UID: \"39c508a1-66a8-4b0e-960a-af901334fd55\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"
Jul 28 17:24:43.073259 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.073190    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39c508a1-66a8-4b0e-960a-af901334fd55-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-m7tdn\" (UID: \"39c508a1-66a8-4b0e-960a-af901334fd55\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"
Jul 28 17:24:43.073259 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.073207    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39c508a1-66a8-4b0e-960a-af901334fd55-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-m7tdn\" (UID: \"39c508a1-66a8-4b0e-960a-af901334fd55\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"
Jul 28 17:24:43.073332 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.073271    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-745rm\" (UniqueName: \"kubernetes.io/projected/39c508a1-66a8-4b0e-960a-af901334fd55-kube-api-access-745rm\") pod \"isvc-autogluon-v1-predictor-574b5b7495-m7tdn\" (UID: \"39c508a1-66a8-4b0e-960a-af901334fd55\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"
Jul 28 17:24:43.073370 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.073336    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39c508a1-66a8-4b0e-960a-af901334fd55-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-m7tdn\" (UID: \"39c508a1-66a8-4b0e-960a-af901334fd55\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"
Jul 28 17:24:43.173797 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.173760    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39c508a1-66a8-4b0e-960a-af901334fd55-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-m7tdn\" (UID: \"39c508a1-66a8-4b0e-960a-af901334fd55\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"
Jul 28 17:24:43.173797 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.173801    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39c508a1-66a8-4b0e-960a-af901334fd55-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-m7tdn\" (UID: \"39c508a1-66a8-4b0e-960a-af901334fd55\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"
Jul 28 17:24:43.174062 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.173820    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39c508a1-66a8-4b0e-960a-af901334fd55-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-m7tdn\" (UID: \"39c508a1-66a8-4b0e-960a-af901334fd55\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"
Jul 28 17:24:43.174062 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.173855    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-745rm\" (UniqueName: \"kubernetes.io/projected/39c508a1-66a8-4b0e-960a-af901334fd55-kube-api-access-745rm\") pod \"isvc-autogluon-v1-predictor-574b5b7495-m7tdn\" (UID: \"39c508a1-66a8-4b0e-960a-af901334fd55\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"
Jul 28 17:24:43.174062 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.173894    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39c508a1-66a8-4b0e-960a-af901334fd55-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-m7tdn\" (UID: \"39c508a1-66a8-4b0e-960a-af901334fd55\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"
Jul 28 17:24:43.174366 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.174343    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39c508a1-66a8-4b0e-960a-af901334fd55-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-m7tdn\" (UID: \"39c508a1-66a8-4b0e-960a-af901334fd55\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"
Jul 28 17:24:43.174546 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.174529    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39c508a1-66a8-4b0e-960a-af901334fd55-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-m7tdn\" (UID: \"39c508a1-66a8-4b0e-960a-af901334fd55\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"
Jul 28 17:24:43.174605 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.174544    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39c508a1-66a8-4b0e-960a-af901334fd55-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-m7tdn\" (UID: \"39c508a1-66a8-4b0e-960a-af901334fd55\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"
Jul 28 17:24:43.176102 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.176084    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39c508a1-66a8-4b0e-960a-af901334fd55-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-m7tdn\" (UID: \"39c508a1-66a8-4b0e-960a-af901334fd55\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"
Jul 28 17:24:43.182432 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.182405    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-745rm\" (UniqueName: \"kubernetes.io/projected/39c508a1-66a8-4b0e-960a-af901334fd55-kube-api-access-745rm\") pod \"isvc-autogluon-v1-predictor-574b5b7495-m7tdn\" (UID: \"39c508a1-66a8-4b0e-960a-af901334fd55\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"
Jul 28 17:24:43.197640 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.197623    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj"
Jul 28 17:24:43.274360 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.274333    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/978898e3-c721-458e-8940-7f9b3a28d05b-proxy-tls\") pod \"978898e3-c721-458e-8940-7f9b3a28d05b\" (UID: \"978898e3-c721-458e-8940-7f9b3a28d05b\") "
Jul 28 17:24:43.274510 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.274374    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/978898e3-c721-458e-8940-7f9b3a28d05b-kube-rbac-proxy-sar-config\") pod \"978898e3-c721-458e-8940-7f9b3a28d05b\" (UID: \"978898e3-c721-458e-8940-7f9b3a28d05b\") "
Jul 28 17:24:43.274510 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.274411    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4966\" (UniqueName: \"kubernetes.io/projected/978898e3-c721-458e-8940-7f9b3a28d05b-kube-api-access-t4966\") pod \"978898e3-c721-458e-8940-7f9b3a28d05b\" (UID: \"978898e3-c721-458e-8940-7f9b3a28d05b\") "
Jul 28 17:24:43.274776 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.274744    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978898e3-c721-458e-8940-7f9b3a28d05b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "978898e3-c721-458e-8940-7f9b3a28d05b" (UID: "978898e3-c721-458e-8940-7f9b3a28d05b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:24:43.276403 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.276381    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/978898e3-c721-458e-8940-7f9b3a28d05b-kube-api-access-t4966" (OuterVolumeSpecName: "kube-api-access-t4966") pod "978898e3-c721-458e-8940-7f9b3a28d05b" (UID: "978898e3-c721-458e-8940-7f9b3a28d05b"). InnerVolumeSpecName "kube-api-access-t4966". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:24:43.276476 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.276428    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/978898e3-c721-458e-8940-7f9b3a28d05b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "978898e3-c721-458e-8940-7f9b3a28d05b" (UID: "978898e3-c721-458e-8940-7f9b3a28d05b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:24:43.339865 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.339838    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"
Jul 28 17:24:43.375118 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.375089    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/978898e3-c721-458e-8940-7f9b3a28d05b-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:24:43.375118 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.375119    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/978898e3-c721-458e-8940-7f9b3a28d05b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:24:43.375276 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.375133    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4966\" (UniqueName: \"kubernetes.io/projected/978898e3-c721-458e-8940-7f9b3a28d05b-kube-api-access-t4966\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:24:43.463815 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.463786    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"]
Jul 28 17:24:43.467501 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:24:43.467474    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c508a1_66a8_4b0e_960a_af901334fd55.slice/crio-56739d05009065a36d22a16c30f278ccd6d27374d5876f4a0cb537531321ce37 WatchSource:0}: Error finding container 56739d05009065a36d22a16c30f278ccd6d27374d5876f4a0cb537531321ce37: Status 404 returned error can't find the container with id 56739d05009065a36d22a16c30f278ccd6d27374d5876f4a0cb537531321ce37
Jul 28 17:24:43.536274 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.536243    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn" event={"ID":"39c508a1-66a8-4b0e-960a-af901334fd55","Type":"ContainerStarted","Data":"5b29ae84925d05ba7f5b6385bc0204f76f394c0cd15b2cafe82602da78b27818"}
Jul 28 17:24:43.536386 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.536322    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn" event={"ID":"39c508a1-66a8-4b0e-960a-af901334fd55","Type":"ContainerStarted","Data":"56739d05009065a36d22a16c30f278ccd6d27374d5876f4a0cb537531321ce37"}
Jul 28 17:24:43.537909 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.537879    2581 generic.go:356] "Generic (PLEG): container finished" podID="978898e3-c721-458e-8940-7f9b3a28d05b" containerID="27e3abade736f9d5d4bad1877c2b1eab302bda08118d0ca4f295922ca8363231" exitCode=2
Jul 28 17:24:43.537909 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.537904    2581 generic.go:356] "Generic (PLEG): container finished" podID="978898e3-c721-458e-8940-7f9b3a28d05b" containerID="95d8a205fbff3d17d6b29298763e10ed96ddf7b80a75b36000276d5796b3ca84" exitCode=2
Jul 28 17:24:43.538087 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.537929    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj"
Jul 28 17:24:43.538087 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.537965    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj" event={"ID":"978898e3-c721-458e-8940-7f9b3a28d05b","Type":"ContainerDied","Data":"27e3abade736f9d5d4bad1877c2b1eab302bda08118d0ca4f295922ca8363231"}
Jul 28 17:24:43.538087 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.538019    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj" event={"ID":"978898e3-c721-458e-8940-7f9b3a28d05b","Type":"ContainerDied","Data":"95d8a205fbff3d17d6b29298763e10ed96ddf7b80a75b36000276d5796b3ca84"}
Jul 28 17:24:43.538087 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.538037    2581 scope.go:117] "RemoveContainer" containerID="27e3abade736f9d5d4bad1877c2b1eab302bda08118d0ca4f295922ca8363231"
Jul 28 17:24:43.538087 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.538052    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj" event={"ID":"978898e3-c721-458e-8940-7f9b3a28d05b","Type":"ContainerDied","Data":"45611958eaeece522480cbce04023c34766431f59b7d2e0b3f6af89b4823766b"}
Jul 28 17:24:43.540818 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.540792    2581 generic.go:356] "Generic (PLEG): container finished" podID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerID="b567d0c3c4c4e13a2ac0a54db0e3dba04fd69020d1a3206d9c02045287afc014" exitCode=2
Jul 28 17:24:43.541026 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.540826    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" event={"ID":"48041153-9cc2-409a-b1a9-9cc566c8faec","Type":"ContainerDied","Data":"b567d0c3c4c4e13a2ac0a54db0e3dba04fd69020d1a3206d9c02045287afc014"}
Jul 28 17:24:43.545818 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.545796    2581 scope.go:117] "RemoveContainer" containerID="95d8a205fbff3d17d6b29298763e10ed96ddf7b80a75b36000276d5796b3ca84"
Jul 28 17:24:43.552514 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.552498    2581 scope.go:117] "RemoveContainer" containerID="27e3abade736f9d5d4bad1877c2b1eab302bda08118d0ca4f295922ca8363231"
Jul 28 17:24:43.552753 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:24:43.552735    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27e3abade736f9d5d4bad1877c2b1eab302bda08118d0ca4f295922ca8363231\": container with ID starting with 27e3abade736f9d5d4bad1877c2b1eab302bda08118d0ca4f295922ca8363231 not found: ID does not exist" containerID="27e3abade736f9d5d4bad1877c2b1eab302bda08118d0ca4f295922ca8363231"
Jul 28 17:24:43.552804 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.552761    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27e3abade736f9d5d4bad1877c2b1eab302bda08118d0ca4f295922ca8363231"} err="failed to get container status \"27e3abade736f9d5d4bad1877c2b1eab302bda08118d0ca4f295922ca8363231\": rpc error: code = NotFound desc = could not find container \"27e3abade736f9d5d4bad1877c2b1eab302bda08118d0ca4f295922ca8363231\": container with ID starting with 27e3abade736f9d5d4bad1877c2b1eab302bda08118d0ca4f295922ca8363231 not found: ID does not exist"
Jul 28 17:24:43.552804 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.552776    2581 scope.go:117] "RemoveContainer" containerID="95d8a205fbff3d17d6b29298763e10ed96ddf7b80a75b36000276d5796b3ca84"
Jul 28 17:24:43.552996 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:24:43.552981    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95d8a205fbff3d17d6b29298763e10ed96ddf7b80a75b36000276d5796b3ca84\": container with ID starting with 95d8a205fbff3d17d6b29298763e10ed96ddf7b80a75b36000276d5796b3ca84 not found: ID does not exist" containerID="95d8a205fbff3d17d6b29298763e10ed96ddf7b80a75b36000276d5796b3ca84"
Jul 28 17:24:43.553047 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.552999    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95d8a205fbff3d17d6b29298763e10ed96ddf7b80a75b36000276d5796b3ca84"} err="failed to get container status \"95d8a205fbff3d17d6b29298763e10ed96ddf7b80a75b36000276d5796b3ca84\": rpc error: code = NotFound desc = could not find container \"95d8a205fbff3d17d6b29298763e10ed96ddf7b80a75b36000276d5796b3ca84\": container with ID starting with 95d8a205fbff3d17d6b29298763e10ed96ddf7b80a75b36000276d5796b3ca84 not found: ID does not exist"
Jul 28 17:24:43.553047 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.553011    2581 scope.go:117] "RemoveContainer" containerID="27e3abade736f9d5d4bad1877c2b1eab302bda08118d0ca4f295922ca8363231"
Jul 28 17:24:43.553241 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.553224    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27e3abade736f9d5d4bad1877c2b1eab302bda08118d0ca4f295922ca8363231"} err="failed to get container status \"27e3abade736f9d5d4bad1877c2b1eab302bda08118d0ca4f295922ca8363231\": rpc error: code = NotFound desc = could not find container \"27e3abade736f9d5d4bad1877c2b1eab302bda08118d0ca4f295922ca8363231\": container with ID starting with 27e3abade736f9d5d4bad1877c2b1eab302bda08118d0ca4f295922ca8363231 not found: ID does not exist"
Jul 28 17:24:43.553283 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.553242    2581 scope.go:117] "RemoveContainer" containerID="95d8a205fbff3d17d6b29298763e10ed96ddf7b80a75b36000276d5796b3ca84"
Jul 28 17:24:43.553451 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.553435    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95d8a205fbff3d17d6b29298763e10ed96ddf7b80a75b36000276d5796b3ca84"} err="failed to get container status \"95d8a205fbff3d17d6b29298763e10ed96ddf7b80a75b36000276d5796b3ca84\": rpc error: code = NotFound desc = could not find container \"95d8a205fbff3d17d6b29298763e10ed96ddf7b80a75b36000276d5796b3ca84\": container with ID starting with 95d8a205fbff3d17d6b29298763e10ed96ddf7b80a75b36000276d5796b3ca84 not found: ID does not exist"
Jul 28 17:24:43.570298 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.570274    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj"]
Jul 28 17:24:43.574761 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:43.574736    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj"]
Jul 28 17:24:44.130573 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:44.130461    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-f9dbj" podUID="978898e3-c721-458e-8940-7f9b3a28d05b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.19:8643/healthz\": context deadline exceeded"
Jul 28 17:24:44.289365 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:44.289331    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="978898e3-c721-458e-8940-7f9b3a28d05b" path="/var/lib/kubelet/pods/978898e3-c721-458e-8940-7f9b3a28d05b/volumes"
Jul 28 17:24:44.664166 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:44.664129    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:24:45.218194 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:45.218149    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Jul 28 17:24:47.557344 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:47.557315    2581 generic.go:356] "Generic (PLEG): container finished" podID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerID="07d3499eef83bf82b79dce485bcf0742a96e838a3342ff3a5819ce51a4cd6902" exitCode=0
Jul 28 17:24:47.557702 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:47.557388    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" event={"ID":"48041153-9cc2-409a-b1a9-9cc566c8faec","Type":"ContainerDied","Data":"07d3499eef83bf82b79dce485bcf0742a96e838a3342ff3a5819ce51a4cd6902"}
Jul 28 17:24:50.218457 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:50.218415    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Jul 28 17:24:50.222777 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:50.222746    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 28 17:24:50.223118 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:50.223091    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:24:55.218152 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:55.218110    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Jul 28 17:24:55.218522 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:24:55.218230    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"
Jul 28 17:25:00.218351 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:00.218308    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Jul 28 17:25:00.222625 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:00.222590    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 28 17:25:00.222901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:00.222874    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:25:05.218686 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:05.218642    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Jul 28 17:25:10.218925 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:10.218880    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.20:8643/healthz\": dial tcp 10.133.0.20:8643: connect: connection refused"
Jul 28 17:25:10.223203 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:10.223165    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.20:8080: connect: connection refused"
Jul 28 17:25:10.223365 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:10.223349    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"
Jul 28 17:25:10.223510 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:10.223493    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:25:10.223615 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:10.223601    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"
Jul 28 17:25:13.006351 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.006327    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"
Jul 28 17:25:13.085753 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.085721    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2b8n\" (UniqueName: \"kubernetes.io/projected/48041153-9cc2-409a-b1a9-9cc566c8faec-kube-api-access-l2b8n\") pod \"48041153-9cc2-409a-b1a9-9cc566c8faec\" (UID: \"48041153-9cc2-409a-b1a9-9cc566c8faec\") "
Jul 28 17:25:13.085896 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.085799    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48041153-9cc2-409a-b1a9-9cc566c8faec-kserve-provision-location\") pod \"48041153-9cc2-409a-b1a9-9cc566c8faec\" (UID: \"48041153-9cc2-409a-b1a9-9cc566c8faec\") "
Jul 28 17:25:13.085896 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.085831    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48041153-9cc2-409a-b1a9-9cc566c8faec-cabundle-cert\") pod \"48041153-9cc2-409a-b1a9-9cc566c8faec\" (UID: \"48041153-9cc2-409a-b1a9-9cc566c8faec\") "
Jul 28 17:25:13.085896 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.085868    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48041153-9cc2-409a-b1a9-9cc566c8faec-kube-rbac-proxy-sar-config\") pod \"48041153-9cc2-409a-b1a9-9cc566c8faec\" (UID: \"48041153-9cc2-409a-b1a9-9cc566c8faec\") "
Jul 28 17:25:13.086080 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.085901    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48041153-9cc2-409a-b1a9-9cc566c8faec-proxy-tls\") pod \"48041153-9cc2-409a-b1a9-9cc566c8faec\" (UID: \"48041153-9cc2-409a-b1a9-9cc566c8faec\") "
Jul 28 17:25:13.086198 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.086170    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48041153-9cc2-409a-b1a9-9cc566c8faec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "48041153-9cc2-409a-b1a9-9cc566c8faec" (UID: "48041153-9cc2-409a-b1a9-9cc566c8faec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:25:13.086299 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.086177    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48041153-9cc2-409a-b1a9-9cc566c8faec-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "48041153-9cc2-409a-b1a9-9cc566c8faec" (UID: "48041153-9cc2-409a-b1a9-9cc566c8faec"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:25:13.086299 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.086248    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48041153-9cc2-409a-b1a9-9cc566c8faec-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "48041153-9cc2-409a-b1a9-9cc566c8faec" (UID: "48041153-9cc2-409a-b1a9-9cc566c8faec"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:25:13.087803 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.087782    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48041153-9cc2-409a-b1a9-9cc566c8faec-kube-api-access-l2b8n" (OuterVolumeSpecName: "kube-api-access-l2b8n") pod "48041153-9cc2-409a-b1a9-9cc566c8faec" (UID: "48041153-9cc2-409a-b1a9-9cc566c8faec"). InnerVolumeSpecName "kube-api-access-l2b8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:25:13.088057 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.088029    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48041153-9cc2-409a-b1a9-9cc566c8faec-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "48041153-9cc2-409a-b1a9-9cc566c8faec" (UID: "48041153-9cc2-409a-b1a9-9cc566c8faec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:25:13.186634 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.186609    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48041153-9cc2-409a-b1a9-9cc566c8faec-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:25:13.186634 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.186632    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48041153-9cc2-409a-b1a9-9cc566c8faec-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:25:13.186776 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.186649    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2b8n\" (UniqueName: \"kubernetes.io/projected/48041153-9cc2-409a-b1a9-9cc566c8faec-kube-api-access-l2b8n\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:25:13.186776 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.186663    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48041153-9cc2-409a-b1a9-9cc566c8faec-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:25:13.186776 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.186676    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48041153-9cc2-409a-b1a9-9cc566c8faec-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:25:13.650057 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.650027    2581 generic.go:356] "Generic (PLEG): container finished" podID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerID="c58a59734988ace78c444b35d2576327bd8aacd134dc89846b2648c3837117c0" exitCode=0
Jul 28 17:25:13.650235 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.650103    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"
Jul 28 17:25:13.650235 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.650106    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" event={"ID":"48041153-9cc2-409a-b1a9-9cc566c8faec","Type":"ContainerDied","Data":"c58a59734988ace78c444b35d2576327bd8aacd134dc89846b2648c3837117c0"}
Jul 28 17:25:13.650354 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.650239    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf" event={"ID":"48041153-9cc2-409a-b1a9-9cc566c8faec","Type":"ContainerDied","Data":"17bc9befd7d0cefa67d908f9b4abaf75df8c66aa66a31332653447e47601b6ee"}
Jul 28 17:25:13.650354 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.650259    2581 scope.go:117] "RemoveContainer" containerID="c58a59734988ace78c444b35d2576327bd8aacd134dc89846b2648c3837117c0"
Jul 28 17:25:13.651756 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.651734    2581 generic.go:356] "Generic (PLEG): container finished" podID="39c508a1-66a8-4b0e-960a-af901334fd55" containerID="5b29ae84925d05ba7f5b6385bc0204f76f394c0cd15b2cafe82602da78b27818" exitCode=0
Jul 28 17:25:13.651836 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.651786    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn" event={"ID":"39c508a1-66a8-4b0e-960a-af901334fd55","Type":"ContainerDied","Data":"5b29ae84925d05ba7f5b6385bc0204f76f394c0cd15b2cafe82602da78b27818"}
Jul 28 17:25:13.658342 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.658326    2581 scope.go:117] "RemoveContainer" containerID="b567d0c3c4c4e13a2ac0a54db0e3dba04fd69020d1a3206d9c02045287afc014"
Jul 28 17:25:13.667476 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.667457    2581 scope.go:117] "RemoveContainer" containerID="07d3499eef83bf82b79dce485bcf0742a96e838a3342ff3a5819ce51a4cd6902"
Jul 28 17:25:13.674002 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.673983    2581 scope.go:117] "RemoveContainer" containerID="844f25c5241885e6421fbab88b63c82d818ce559dd26bceb6bb60ce179ca5858"
Jul 28 17:25:13.680384 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.680367    2581 scope.go:117] "RemoveContainer" containerID="c58a59734988ace78c444b35d2576327bd8aacd134dc89846b2648c3837117c0"
Jul 28 17:25:13.680647 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:25:13.680628    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c58a59734988ace78c444b35d2576327bd8aacd134dc89846b2648c3837117c0\": container with ID starting with c58a59734988ace78c444b35d2576327bd8aacd134dc89846b2648c3837117c0 not found: ID does not exist" containerID="c58a59734988ace78c444b35d2576327bd8aacd134dc89846b2648c3837117c0"
Jul 28 17:25:13.680694 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.680654    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c58a59734988ace78c444b35d2576327bd8aacd134dc89846b2648c3837117c0"} err="failed to get container status \"c58a59734988ace78c444b35d2576327bd8aacd134dc89846b2648c3837117c0\": rpc error: code = NotFound desc = could not find container \"c58a59734988ace78c444b35d2576327bd8aacd134dc89846b2648c3837117c0\": container with ID starting with c58a59734988ace78c444b35d2576327bd8aacd134dc89846b2648c3837117c0 not found: ID does not exist"
Jul 28 17:25:13.680694 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.680671    2581 scope.go:117] "RemoveContainer" containerID="b567d0c3c4c4e13a2ac0a54db0e3dba04fd69020d1a3206d9c02045287afc014"
Jul 28 17:25:13.680989 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:25:13.680928    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b567d0c3c4c4e13a2ac0a54db0e3dba04fd69020d1a3206d9c02045287afc014\": container with ID starting with b567d0c3c4c4e13a2ac0a54db0e3dba04fd69020d1a3206d9c02045287afc014 not found: ID does not exist" containerID="b567d0c3c4c4e13a2ac0a54db0e3dba04fd69020d1a3206d9c02045287afc014"
Jul 28 17:25:13.680989 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.680975    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b567d0c3c4c4e13a2ac0a54db0e3dba04fd69020d1a3206d9c02045287afc014"} err="failed to get container status \"b567d0c3c4c4e13a2ac0a54db0e3dba04fd69020d1a3206d9c02045287afc014\": rpc error: code = NotFound desc = could not find container \"b567d0c3c4c4e13a2ac0a54db0e3dba04fd69020d1a3206d9c02045287afc014\": container with ID starting with b567d0c3c4c4e13a2ac0a54db0e3dba04fd69020d1a3206d9c02045287afc014 not found: ID does not exist"
Jul 28 17:25:13.680989 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.680995    2581 scope.go:117] "RemoveContainer" containerID="07d3499eef83bf82b79dce485bcf0742a96e838a3342ff3a5819ce51a4cd6902"
Jul 28 17:25:13.681268 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:25:13.681247    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07d3499eef83bf82b79dce485bcf0742a96e838a3342ff3a5819ce51a4cd6902\": container with ID starting with 07d3499eef83bf82b79dce485bcf0742a96e838a3342ff3a5819ce51a4cd6902 not found: ID does not exist" containerID="07d3499eef83bf82b79dce485bcf0742a96e838a3342ff3a5819ce51a4cd6902"
Jul 28 17:25:13.681380 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.681267    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07d3499eef83bf82b79dce485bcf0742a96e838a3342ff3a5819ce51a4cd6902"} err="failed to get container status \"07d3499eef83bf82b79dce485bcf0742a96e838a3342ff3a5819ce51a4cd6902\": rpc error: code = NotFound desc = could not find container \"07d3499eef83bf82b79dce485bcf0742a96e838a3342ff3a5819ce51a4cd6902\": container with ID starting with 07d3499eef83bf82b79dce485bcf0742a96e838a3342ff3a5819ce51a4cd6902 not found: ID does not exist"
Jul 28 17:25:13.681380 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.681281    2581 scope.go:117] "RemoveContainer" containerID="844f25c5241885e6421fbab88b63c82d818ce559dd26bceb6bb60ce179ca5858"
Jul 28 17:25:13.681616 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:25:13.681594    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"844f25c5241885e6421fbab88b63c82d818ce559dd26bceb6bb60ce179ca5858\": container with ID starting with 844f25c5241885e6421fbab88b63c82d818ce559dd26bceb6bb60ce179ca5858 not found: ID does not exist" containerID="844f25c5241885e6421fbab88b63c82d818ce559dd26bceb6bb60ce179ca5858"
Jul 28 17:25:13.681689 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.681620    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"844f25c5241885e6421fbab88b63c82d818ce559dd26bceb6bb60ce179ca5858"} err="failed to get container status \"844f25c5241885e6421fbab88b63c82d818ce559dd26bceb6bb60ce179ca5858\": rpc error: code = NotFound desc = could not find container \"844f25c5241885e6421fbab88b63c82d818ce559dd26bceb6bb60ce179ca5858\": container with ID starting with 844f25c5241885e6421fbab88b63c82d818ce559dd26bceb6bb60ce179ca5858 not found: ID does not exist"
Jul 28 17:25:13.696484 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.696455    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"]
Jul 28 17:25:13.702464 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:13.702444    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-866fbfd695-w6rrf"]
Jul 28 17:25:14.294092 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:25:14.294012    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" path="/var/lib/kubelet/pods/48041153-9cc2-409a-b1a9-9cc566c8faec/volumes"
Jul 28 17:26:32.174474 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:26:32.174436    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:28.165786 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:27:28.164659    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn" event={"ID":"39c508a1-66a8-4b0e-960a-af901334fd55","Type":"ContainerStarted","Data":"77994fb1b8b3b040324c875f333b900da125f98dc37f65211ef2bc04acefb083"}
Jul 28 17:27:28.165786 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:27:28.164729    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn" event={"ID":"39c508a1-66a8-4b0e-960a-af901334fd55","Type":"ContainerStarted","Data":"afc03b1370d349122d10d5024b86d5ab32cd62b6beb8c4383c630ab966f3c68c"}
Jul 28 17:27:28.165786 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:27:28.165680    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"
Jul 28 17:27:28.165786 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:27:28.165726    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"
Jul 28 17:27:28.166905 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:27:28.166664    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn" podUID="39c508a1-66a8-4b0e-960a-af901334fd55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 28 17:27:28.234860 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:27:28.234819    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn" podStartSLOduration=1.201513543 podStartE2EDuration="2m45.234805518s" totalImagesPullingTime="2m14.240957859s" totalInitContainerRuntime="29.792334116s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:27:28.234332392 +0000 UTC m=+1096.508948824" watchObservedRunningTime="2026-07-28 17:27:28.234805518 +0000 UTC m=+1096.509421949"
Jul 28 17:27:29.169062 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:27:29.169024    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn" podUID="39c508a1-66a8-4b0e-960a-af901334fd55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 28 17:27:30.172919 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:27:30.172879    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn" podUID="39c508a1-66a8-4b0e-960a-af901334fd55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 28 17:27:35.178923 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:27:35.178892    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"
Jul 28 17:27:35.179459 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:27:35.179434    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn" podUID="39c508a1-66a8-4b0e-960a-af901334fd55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Jul 28 17:27:38.479671 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:27:38.479632    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:45.180801 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:27:45.180771    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"
Jul 28 17:27:59.114502 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:27:59.114470    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"]
Jul 28 17:27:59.115099 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:27:59.114746    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn" podUID="39c508a1-66a8-4b0e-960a-af901334fd55" containerName="kserve-container" containerID="cri-o://afc03b1370d349122d10d5024b86d5ab32cd62b6beb8c4383c630ab966f3c68c" gracePeriod=30
Jul 28 17:27:59.115099 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:27:59.114780    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn" podUID="39c508a1-66a8-4b0e-960a-af901334fd55" containerName="kube-rbac-proxy" containerID="cri-o://77994fb1b8b3b040324c875f333b900da125f98dc37f65211ef2bc04acefb083" gracePeriod=30
Jul 28 17:27:59.126320 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:27:59.126295    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:27:59.278868 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:27:59.278838    2581 generic.go:356] "Generic (PLEG): container finished" podID="39c508a1-66a8-4b0e-960a-af901334fd55" containerID="77994fb1b8b3b040324c875f333b900da125f98dc37f65211ef2bc04acefb083" exitCode=2
Jul 28 17:27:59.278996 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:27:59.278916    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn" event={"ID":"39c508a1-66a8-4b0e-960a-af901334fd55","Type":"ContainerDied","Data":"77994fb1b8b3b040324c875f333b900da125f98dc37f65211ef2bc04acefb083"}
Jul 28 17:28:00.174238 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:00.174197    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn" podUID="39c508a1-66a8-4b0e-960a-af901334fd55" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Jul 28 17:28:00.490285 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:00.490260    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"
Jul 28 17:28:00.494458 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:00.494440    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39c508a1-66a8-4b0e-960a-af901334fd55-kserve-provision-location\") pod \"39c508a1-66a8-4b0e-960a-af901334fd55\" (UID: \"39c508a1-66a8-4b0e-960a-af901334fd55\") "
Jul 28 17:28:00.494518 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:00.494469    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-745rm\" (UniqueName: \"kubernetes.io/projected/39c508a1-66a8-4b0e-960a-af901334fd55-kube-api-access-745rm\") pod \"39c508a1-66a8-4b0e-960a-af901334fd55\" (UID: \"39c508a1-66a8-4b0e-960a-af901334fd55\") "
Jul 28 17:28:00.494518 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:00.494498    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39c508a1-66a8-4b0e-960a-af901334fd55-proxy-tls\") pod \"39c508a1-66a8-4b0e-960a-af901334fd55\" (UID: \"39c508a1-66a8-4b0e-960a-af901334fd55\") "
Jul 28 17:28:00.494518 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:00.494513    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39c508a1-66a8-4b0e-960a-af901334fd55-cabundle-cert\") pod \"39c508a1-66a8-4b0e-960a-af901334fd55\" (UID: \"39c508a1-66a8-4b0e-960a-af901334fd55\") "
Jul 28 17:28:00.494658 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:00.494638    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39c508a1-66a8-4b0e-960a-af901334fd55-kube-rbac-proxy-sar-config\") pod \"39c508a1-66a8-4b0e-960a-af901334fd55\" (UID: \"39c508a1-66a8-4b0e-960a-af901334fd55\") "
Jul 28 17:28:00.494936 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:00.494825    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c508a1-66a8-4b0e-960a-af901334fd55-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "39c508a1-66a8-4b0e-960a-af901334fd55" (UID: "39c508a1-66a8-4b0e-960a-af901334fd55"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:28:00.495050 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:00.494984    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c508a1-66a8-4b0e-960a-af901334fd55-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "39c508a1-66a8-4b0e-960a-af901334fd55" (UID: "39c508a1-66a8-4b0e-960a-af901334fd55"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:28:00.496625 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:00.496602    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c508a1-66a8-4b0e-960a-af901334fd55-kube-api-access-745rm" (OuterVolumeSpecName: "kube-api-access-745rm") pod "39c508a1-66a8-4b0e-960a-af901334fd55" (UID: "39c508a1-66a8-4b0e-960a-af901334fd55"). InnerVolumeSpecName "kube-api-access-745rm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:28:00.496700 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:00.496638    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39c508a1-66a8-4b0e-960a-af901334fd55-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "39c508a1-66a8-4b0e-960a-af901334fd55" (UID: "39c508a1-66a8-4b0e-960a-af901334fd55"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:28:00.499311 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:00.499287    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39c508a1-66a8-4b0e-960a-af901334fd55-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "39c508a1-66a8-4b0e-960a-af901334fd55" (UID: "39c508a1-66a8-4b0e-960a-af901334fd55"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:28:00.595914 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:00.595892    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39c508a1-66a8-4b0e-960a-af901334fd55-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:28:00.595914 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:00.595912    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39c508a1-66a8-4b0e-960a-af901334fd55-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:28:00.596068 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:00.595920    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39c508a1-66a8-4b0e-960a-af901334fd55-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:28:00.596068 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:00.595929    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39c508a1-66a8-4b0e-960a-af901334fd55-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:28:00.596068 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:00.595938    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-745rm\" (UniqueName: \"kubernetes.io/projected/39c508a1-66a8-4b0e-960a-af901334fd55-kube-api-access-745rm\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:28:01.286634 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:01.286595    2581 generic.go:356] "Generic (PLEG): container finished" podID="39c508a1-66a8-4b0e-960a-af901334fd55" containerID="afc03b1370d349122d10d5024b86d5ab32cd62b6beb8c4383c630ab966f3c68c" exitCode=0
Jul 28 17:28:01.287142 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:01.286670    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"
Jul 28 17:28:01.287142 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:01.286677    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn" event={"ID":"39c508a1-66a8-4b0e-960a-af901334fd55","Type":"ContainerDied","Data":"afc03b1370d349122d10d5024b86d5ab32cd62b6beb8c4383c630ab966f3c68c"}
Jul 28 17:28:01.287142 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:01.286734    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn" event={"ID":"39c508a1-66a8-4b0e-960a-af901334fd55","Type":"ContainerDied","Data":"56739d05009065a36d22a16c30f278ccd6d27374d5876f4a0cb537531321ce37"}
Jul 28 17:28:01.287142 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:01.286761    2581 scope.go:117] "RemoveContainer" containerID="77994fb1b8b3b040324c875f333b900da125f98dc37f65211ef2bc04acefb083"
Jul 28 17:28:01.295134 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:01.295115    2581 scope.go:117] "RemoveContainer" containerID="afc03b1370d349122d10d5024b86d5ab32cd62b6beb8c4383c630ab966f3c68c"
Jul 28 17:28:01.303145 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:01.303129    2581 scope.go:117] "RemoveContainer" containerID="5b29ae84925d05ba7f5b6385bc0204f76f394c0cd15b2cafe82602da78b27818"
Jul 28 17:28:01.310053 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:01.309955    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"]
Jul 28 17:28:01.310118 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:01.310066    2581 scope.go:117] "RemoveContainer" containerID="77994fb1b8b3b040324c875f333b900da125f98dc37f65211ef2bc04acefb083"
Jul 28 17:28:01.310323 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:28:01.310303    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77994fb1b8b3b040324c875f333b900da125f98dc37f65211ef2bc04acefb083\": container with ID starting with 77994fb1b8b3b040324c875f333b900da125f98dc37f65211ef2bc04acefb083 not found: ID does not exist" containerID="77994fb1b8b3b040324c875f333b900da125f98dc37f65211ef2bc04acefb083"
Jul 28 17:28:01.310382 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:01.310328    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77994fb1b8b3b040324c875f333b900da125f98dc37f65211ef2bc04acefb083"} err="failed to get container status \"77994fb1b8b3b040324c875f333b900da125f98dc37f65211ef2bc04acefb083\": rpc error: code = NotFound desc = could not find container \"77994fb1b8b3b040324c875f333b900da125f98dc37f65211ef2bc04acefb083\": container with ID starting with 77994fb1b8b3b040324c875f333b900da125f98dc37f65211ef2bc04acefb083 not found: ID does not exist"
Jul 28 17:28:01.310382 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:01.310352    2581 scope.go:117] "RemoveContainer" containerID="afc03b1370d349122d10d5024b86d5ab32cd62b6beb8c4383c630ab966f3c68c"
Jul 28 17:28:01.310582 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:28:01.310564    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afc03b1370d349122d10d5024b86d5ab32cd62b6beb8c4383c630ab966f3c68c\": container with ID starting with afc03b1370d349122d10d5024b86d5ab32cd62b6beb8c4383c630ab966f3c68c not found: ID does not exist" containerID="afc03b1370d349122d10d5024b86d5ab32cd62b6beb8c4383c630ab966f3c68c"
Jul 28 17:28:01.310634 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:01.310586    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc03b1370d349122d10d5024b86d5ab32cd62b6beb8c4383c630ab966f3c68c"} err="failed to get container status \"afc03b1370d349122d10d5024b86d5ab32cd62b6beb8c4383c630ab966f3c68c\": rpc error: code = NotFound desc = could not find container \"afc03b1370d349122d10d5024b86d5ab32cd62b6beb8c4383c630ab966f3c68c\": container with ID starting with afc03b1370d349122d10d5024b86d5ab32cd62b6beb8c4383c630ab966f3c68c not found: ID does not exist"
Jul 28 17:28:01.310634 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:01.310600    2581 scope.go:117] "RemoveContainer" containerID="5b29ae84925d05ba7f5b6385bc0204f76f394c0cd15b2cafe82602da78b27818"
Jul 28 17:28:01.310833 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:28:01.310816    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b29ae84925d05ba7f5b6385bc0204f76f394c0cd15b2cafe82602da78b27818\": container with ID starting with 5b29ae84925d05ba7f5b6385bc0204f76f394c0cd15b2cafe82602da78b27818 not found: ID does not exist" containerID="5b29ae84925d05ba7f5b6385bc0204f76f394c0cd15b2cafe82602da78b27818"
Jul 28 17:28:01.310888 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:01.310835    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b29ae84925d05ba7f5b6385bc0204f76f394c0cd15b2cafe82602da78b27818"} err="failed to get container status \"5b29ae84925d05ba7f5b6385bc0204f76f394c0cd15b2cafe82602da78b27818\": rpc error: code = NotFound desc = could not find container \"5b29ae84925d05ba7f5b6385bc0204f76f394c0cd15b2cafe82602da78b27818\": container with ID starting with 5b29ae84925d05ba7f5b6385bc0204f76f394c0cd15b2cafe82602da78b27818 not found: ID does not exist"
Jul 28 17:28:01.314297 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:01.314276    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-m7tdn"]
Jul 28 17:28:02.289846 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:02.289812    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39c508a1-66a8-4b0e-960a-af901334fd55" path="/var/lib/kubelet/pods/39c508a1-66a8-4b0e-960a-af901334fd55/volumes"
Jul 28 17:28:04.313460 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.313421    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:28:04.391372 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.391343    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"]
Jul 28 17:28:04.391584 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.391573    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="agent"
Jul 28 17:28:04.391625 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.391586    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="agent"
Jul 28 17:28:04.391625 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.391599    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="storage-initializer"
Jul 28 17:28:04.391625 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.391604    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="storage-initializer"
Jul 28 17:28:04.391625 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.391612    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="978898e3-c721-458e-8940-7f9b3a28d05b" containerName="kserve-container"
Jul 28 17:28:04.391625 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.391617    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="978898e3-c721-458e-8940-7f9b3a28d05b" containerName="kserve-container"
Jul 28 17:28:04.391625 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.391624    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="kube-rbac-proxy"
Jul 28 17:28:04.391798 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.391630    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="kube-rbac-proxy"
Jul 28 17:28:04.391798 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.391637    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="kserve-container"
Jul 28 17:28:04.391798 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.391642    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="kserve-container"
Jul 28 17:28:04.391798 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.391651    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39c508a1-66a8-4b0e-960a-af901334fd55" containerName="kube-rbac-proxy"
Jul 28 17:28:04.391798 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.391656    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="39c508a1-66a8-4b0e-960a-af901334fd55" containerName="kube-rbac-proxy"
Jul 28 17:28:04.391798 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.391663    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39c508a1-66a8-4b0e-960a-af901334fd55" containerName="storage-initializer"
Jul 28 17:28:04.391798 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.391668    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="39c508a1-66a8-4b0e-960a-af901334fd55" containerName="storage-initializer"
Jul 28 17:28:04.391798 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.391674    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="978898e3-c721-458e-8940-7f9b3a28d05b" containerName="kube-rbac-proxy"
Jul 28 17:28:04.391798 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.391679    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="978898e3-c721-458e-8940-7f9b3a28d05b" containerName="kube-rbac-proxy"
Jul 28 17:28:04.391798 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.391686    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39c508a1-66a8-4b0e-960a-af901334fd55" containerName="kserve-container"
Jul 28 17:28:04.391798 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.391691    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="39c508a1-66a8-4b0e-960a-af901334fd55" containerName="kserve-container"
Jul 28 17:28:04.391798 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.391727    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="39c508a1-66a8-4b0e-960a-af901334fd55" containerName="kserve-container"
Jul 28 17:28:04.391798 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.391735    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="kserve-container"
Jul 28 17:28:04.391798 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.391741    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="kube-rbac-proxy"
Jul 28 17:28:04.391798 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.391747    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="978898e3-c721-458e-8940-7f9b3a28d05b" containerName="kube-rbac-proxy"
Jul 28 17:28:04.391798 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.391753    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="978898e3-c721-458e-8940-7f9b3a28d05b" containerName="kserve-container"
Jul 28 17:28:04.391798 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.391758    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="48041153-9cc2-409a-b1a9-9cc566c8faec" containerName="agent"
Jul 28 17:28:04.391798 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.391763    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="39c508a1-66a8-4b0e-960a-af901334fd55" containerName="kube-rbac-proxy"
Jul 28 17:28:04.422484 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.422455    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"]
Jul 28 17:28:04.422631 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.422551    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"
Jul 28 17:28:04.425565 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.425537    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:28:04.425565 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.425553    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 28 17:28:04.425753 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.425643    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 28 17:28:04.425818 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.425804    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:28:04.427030 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.426999    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:28:04.427134 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.427033    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 17:28:04.427134 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.427098    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:28:04.473974 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.473924    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:28:04.521099 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.521078    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b21e273d-207d-419a-a0e6-b9954be290df-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xcl6d\" (UID: \"b21e273d-207d-419a-a0e6-b9954be290df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"
Jul 28 17:28:04.521203 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.521115    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b21e273d-207d-419a-a0e6-b9954be290df-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xcl6d\" (UID: \"b21e273d-207d-419a-a0e6-b9954be290df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"
Jul 28 17:28:04.521203 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.521197    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfngd\" (UniqueName: \"kubernetes.io/projected/b21e273d-207d-419a-a0e6-b9954be290df-kube-api-access-pfngd\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xcl6d\" (UID: \"b21e273d-207d-419a-a0e6-b9954be290df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"
Jul 28 17:28:04.521316 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.521231    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b21e273d-207d-419a-a0e6-b9954be290df-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xcl6d\" (UID: \"b21e273d-207d-419a-a0e6-b9954be290df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"
Jul 28 17:28:04.521316 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.521285    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b21e273d-207d-419a-a0e6-b9954be290df-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xcl6d\" (UID: \"b21e273d-207d-419a-a0e6-b9954be290df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"
Jul 28 17:28:04.622029 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.621974    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfngd\" (UniqueName: \"kubernetes.io/projected/b21e273d-207d-419a-a0e6-b9954be290df-kube-api-access-pfngd\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xcl6d\" (UID: \"b21e273d-207d-419a-a0e6-b9954be290df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"
Jul 28 17:28:04.622029 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.622003    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b21e273d-207d-419a-a0e6-b9954be290df-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xcl6d\" (UID: \"b21e273d-207d-419a-a0e6-b9954be290df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"
Jul 28 17:28:04.622181 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.622030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b21e273d-207d-419a-a0e6-b9954be290df-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xcl6d\" (UID: \"b21e273d-207d-419a-a0e6-b9954be290df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"
Jul 28 17:28:04.622181 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.622069    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b21e273d-207d-419a-a0e6-b9954be290df-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xcl6d\" (UID: \"b21e273d-207d-419a-a0e6-b9954be290df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"
Jul 28 17:28:04.622181 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.622087    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b21e273d-207d-419a-a0e6-b9954be290df-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xcl6d\" (UID: \"b21e273d-207d-419a-a0e6-b9954be290df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"
Jul 28 17:28:04.622401 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.622382    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b21e273d-207d-419a-a0e6-b9954be290df-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xcl6d\" (UID: \"b21e273d-207d-419a-a0e6-b9954be290df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"
Jul 28 17:28:04.622731 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.622709    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b21e273d-207d-419a-a0e6-b9954be290df-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xcl6d\" (UID: \"b21e273d-207d-419a-a0e6-b9954be290df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"
Jul 28 17:28:04.622794 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.622736    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b21e273d-207d-419a-a0e6-b9954be290df-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xcl6d\" (UID: \"b21e273d-207d-419a-a0e6-b9954be290df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"
Jul 28 17:28:04.624409 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.624392    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b21e273d-207d-419a-a0e6-b9954be290df-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xcl6d\" (UID: \"b21e273d-207d-419a-a0e6-b9954be290df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"
Jul 28 17:28:04.632573 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.632550    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfngd\" (UniqueName: \"kubernetes.io/projected/b21e273d-207d-419a-a0e6-b9954be290df-kube-api-access-pfngd\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-xcl6d\" (UID: \"b21e273d-207d-419a-a0e6-b9954be290df\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"
Jul 28 17:28:04.733740 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.733712    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"
Jul 28 17:28:04.879482 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:04.879371    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"]
Jul 28 17:28:04.881768 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:28:04.881743    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb21e273d_207d_419a_a0e6_b9954be290df.slice/crio-28257dae0ef91e6d5f95300ae391805bc426e64d6da46973023420f9f55849cc WatchSource:0}: Error finding container 28257dae0ef91e6d5f95300ae391805bc426e64d6da46973023420f9f55849cc: Status 404 returned error can't find the container with id 28257dae0ef91e6d5f95300ae391805bc426e64d6da46973023420f9f55849cc
Jul 28 17:28:05.303602 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:05.303564    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d" event={"ID":"b21e273d-207d-419a-a0e6-b9954be290df","Type":"ContainerStarted","Data":"b1dafa4b4fa23e4698de893cf940341ef80a7863505b11767453855fd3c5150f"}
Jul 28 17:28:05.303796 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:05.303648    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d" event={"ID":"b21e273d-207d-419a-a0e6-b9954be290df","Type":"ContainerStarted","Data":"28257dae0ef91e6d5f95300ae391805bc426e64d6da46973023420f9f55849cc"}
Jul 28 17:28:10.321218 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:10.321187    2581 generic.go:356] "Generic (PLEG): container finished" podID="b21e273d-207d-419a-a0e6-b9954be290df" containerID="b1dafa4b4fa23e4698de893cf940341ef80a7863505b11767453855fd3c5150f" exitCode=0
Jul 28 17:28:10.321607 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:10.321260    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d" event={"ID":"b21e273d-207d-419a-a0e6-b9954be290df","Type":"ContainerDied","Data":"b1dafa4b4fa23e4698de893cf940341ef80a7863505b11767453855fd3c5150f"}
Jul 28 17:28:11.326986 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:11.326930    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d" event={"ID":"b21e273d-207d-419a-a0e6-b9954be290df","Type":"ContainerStarted","Data":"3485521dafd6772b2399f8eeb7659b143db8a798ba71318f09f98f79a768abfc"}
Jul 28 17:28:11.327426 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:11.327010    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d" event={"ID":"b21e273d-207d-419a-a0e6-b9954be290df","Type":"ContainerStarted","Data":"b3125dac3c3ca47c69495c2dbf80117e44220466994d393e6f97891f3beb73c3"}
Jul 28 17:28:11.327426 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:11.327353    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"
Jul 28 17:28:11.327516 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:11.327434    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"
Jul 28 17:28:11.350868 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:11.350809    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d" podStartSLOduration=2.269800252 podStartE2EDuration="7.350788639s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.080988387s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:28:11.347904695 +0000 UTC m=+1139.622521126" watchObservedRunningTime="2026-07-28 17:28:11.350788639 +0000 UTC m=+1139.625405072"
Jul 28 17:28:17.336537 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:28:17.336507    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"
Jul 28 17:29:12.235493 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:29:12.235462    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 17:29:12.236078 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:29:12.235780    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 17:29:44.289520 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:29:44.289484    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"
Jul 28 17:29:55.327203 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:29:55.327148    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d" podUID="b21e273d-207d-419a-a0e6-b9954be290df" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.22:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 17:30:00.387227 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:00.387181    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:00.455545 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:00.455515    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"]
Jul 28 17:30:00.455802 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:00.455777    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d" podUID="b21e273d-207d-419a-a0e6-b9954be290df" containerName="kserve-container" containerID="cri-o://b3125dac3c3ca47c69495c2dbf80117e44220466994d393e6f97891f3beb73c3" gracePeriod=30
Jul 28 17:30:00.455887 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:00.455834    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d" podUID="b21e273d-207d-419a-a0e6-b9954be290df" containerName="kube-rbac-proxy" containerID="cri-o://3485521dafd6772b2399f8eeb7659b143db8a798ba71318f09f98f79a768abfc" gracePeriod=30
Jul 28 17:30:00.701513 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:00.701474    2581 generic.go:356] "Generic (PLEG): container finished" podID="b21e273d-207d-419a-a0e6-b9954be290df" containerID="3485521dafd6772b2399f8eeb7659b143db8a798ba71318f09f98f79a768abfc" exitCode=2
Jul 28 17:30:00.701682 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:00.701545    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d" event={"ID":"b21e273d-207d-419a-a0e6-b9954be290df","Type":"ContainerDied","Data":"3485521dafd6772b2399f8eeb7659b143db8a798ba71318f09f98f79a768abfc"}
Jul 28 17:30:01.694831 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.694808    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"
Jul 28 17:30:01.706084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.706057    2581 generic.go:356] "Generic (PLEG): container finished" podID="b21e273d-207d-419a-a0e6-b9954be290df" containerID="b3125dac3c3ca47c69495c2dbf80117e44220466994d393e6f97891f3beb73c3" exitCode=0
Jul 28 17:30:01.706198 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.706109    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"
Jul 28 17:30:01.706198 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.706146    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d" event={"ID":"b21e273d-207d-419a-a0e6-b9954be290df","Type":"ContainerDied","Data":"b3125dac3c3ca47c69495c2dbf80117e44220466994d393e6f97891f3beb73c3"}
Jul 28 17:30:01.706317 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.706214    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d" event={"ID":"b21e273d-207d-419a-a0e6-b9954be290df","Type":"ContainerDied","Data":"28257dae0ef91e6d5f95300ae391805bc426e64d6da46973023420f9f55849cc"}
Jul 28 17:30:01.706317 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.706256    2581 scope.go:117] "RemoveContainer" containerID="3485521dafd6772b2399f8eeb7659b143db8a798ba71318f09f98f79a768abfc"
Jul 28 17:30:01.713380 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.713365    2581 scope.go:117] "RemoveContainer" containerID="b3125dac3c3ca47c69495c2dbf80117e44220466994d393e6f97891f3beb73c3"
Jul 28 17:30:01.719831 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.719816    2581 scope.go:117] "RemoveContainer" containerID="b1dafa4b4fa23e4698de893cf940341ef80a7863505b11767453855fd3c5150f"
Jul 28 17:30:01.727393 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.727360    2581 scope.go:117] "RemoveContainer" containerID="3485521dafd6772b2399f8eeb7659b143db8a798ba71318f09f98f79a768abfc"
Jul 28 17:30:01.727675 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:30:01.727648    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3485521dafd6772b2399f8eeb7659b143db8a798ba71318f09f98f79a768abfc\": container with ID starting with 3485521dafd6772b2399f8eeb7659b143db8a798ba71318f09f98f79a768abfc not found: ID does not exist" containerID="3485521dafd6772b2399f8eeb7659b143db8a798ba71318f09f98f79a768abfc"
Jul 28 17:30:01.727758 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.727682    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3485521dafd6772b2399f8eeb7659b143db8a798ba71318f09f98f79a768abfc"} err="failed to get container status \"3485521dafd6772b2399f8eeb7659b143db8a798ba71318f09f98f79a768abfc\": rpc error: code = NotFound desc = could not find container \"3485521dafd6772b2399f8eeb7659b143db8a798ba71318f09f98f79a768abfc\": container with ID starting with 3485521dafd6772b2399f8eeb7659b143db8a798ba71318f09f98f79a768abfc not found: ID does not exist"
Jul 28 17:30:01.727758 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.727705    2581 scope.go:117] "RemoveContainer" containerID="b3125dac3c3ca47c69495c2dbf80117e44220466994d393e6f97891f3beb73c3"
Jul 28 17:30:01.727980 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:30:01.727961    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3125dac3c3ca47c69495c2dbf80117e44220466994d393e6f97891f3beb73c3\": container with ID starting with b3125dac3c3ca47c69495c2dbf80117e44220466994d393e6f97891f3beb73c3 not found: ID does not exist" containerID="b3125dac3c3ca47c69495c2dbf80117e44220466994d393e6f97891f3beb73c3"
Jul 28 17:30:01.728061 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.727989    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3125dac3c3ca47c69495c2dbf80117e44220466994d393e6f97891f3beb73c3"} err="failed to get container status \"b3125dac3c3ca47c69495c2dbf80117e44220466994d393e6f97891f3beb73c3\": rpc error: code = NotFound desc = could not find container \"b3125dac3c3ca47c69495c2dbf80117e44220466994d393e6f97891f3beb73c3\": container with ID starting with b3125dac3c3ca47c69495c2dbf80117e44220466994d393e6f97891f3beb73c3 not found: ID does not exist"
Jul 28 17:30:01.728061 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.728010    2581 scope.go:117] "RemoveContainer" containerID="b1dafa4b4fa23e4698de893cf940341ef80a7863505b11767453855fd3c5150f"
Jul 28 17:30:01.728276 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:30:01.728255    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1dafa4b4fa23e4698de893cf940341ef80a7863505b11767453855fd3c5150f\": container with ID starting with b1dafa4b4fa23e4698de893cf940341ef80a7863505b11767453855fd3c5150f not found: ID does not exist" containerID="b1dafa4b4fa23e4698de893cf940341ef80a7863505b11767453855fd3c5150f"
Jul 28 17:30:01.728350 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.728279    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1dafa4b4fa23e4698de893cf940341ef80a7863505b11767453855fd3c5150f"} err="failed to get container status \"b1dafa4b4fa23e4698de893cf940341ef80a7863505b11767453855fd3c5150f\": rpc error: code = NotFound desc = could not find container \"b1dafa4b4fa23e4698de893cf940341ef80a7863505b11767453855fd3c5150f\": container with ID starting with b1dafa4b4fa23e4698de893cf940341ef80a7863505b11767453855fd3c5150f not found: ID does not exist"
Jul 28 17:30:01.769335 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.769310    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b21e273d-207d-419a-a0e6-b9954be290df-kserve-provision-location\") pod \"b21e273d-207d-419a-a0e6-b9954be290df\" (UID: \"b21e273d-207d-419a-a0e6-b9954be290df\") "
Jul 28 17:30:01.769442 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.769351    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b21e273d-207d-419a-a0e6-b9954be290df-kube-rbac-proxy-sar-config\") pod \"b21e273d-207d-419a-a0e6-b9954be290df\" (UID: \"b21e273d-207d-419a-a0e6-b9954be290df\") "
Jul 28 17:30:01.769442 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.769387    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b21e273d-207d-419a-a0e6-b9954be290df-proxy-tls\") pod \"b21e273d-207d-419a-a0e6-b9954be290df\" (UID: \"b21e273d-207d-419a-a0e6-b9954be290df\") "
Jul 28 17:30:01.769554 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.769441    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b21e273d-207d-419a-a0e6-b9954be290df-cabundle-cert\") pod \"b21e273d-207d-419a-a0e6-b9954be290df\" (UID: \"b21e273d-207d-419a-a0e6-b9954be290df\") "
Jul 28 17:30:01.769554 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.769467    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfngd\" (UniqueName: \"kubernetes.io/projected/b21e273d-207d-419a-a0e6-b9954be290df-kube-api-access-pfngd\") pod \"b21e273d-207d-419a-a0e6-b9954be290df\" (UID: \"b21e273d-207d-419a-a0e6-b9954be290df\") "
Jul 28 17:30:01.769776 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.769745    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b21e273d-207d-419a-a0e6-b9954be290df-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b21e273d-207d-419a-a0e6-b9954be290df" (UID: "b21e273d-207d-419a-a0e6-b9954be290df"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:30:01.769844 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.769788    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b21e273d-207d-419a-a0e6-b9954be290df-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b21e273d-207d-419a-a0e6-b9954be290df" (UID: "b21e273d-207d-419a-a0e6-b9954be290df"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:30:01.771330 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.771306    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b21e273d-207d-419a-a0e6-b9954be290df-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b21e273d-207d-419a-a0e6-b9954be290df" (UID: "b21e273d-207d-419a-a0e6-b9954be290df"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:30:01.771552 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.771519    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b21e273d-207d-419a-a0e6-b9954be290df-kube-api-access-pfngd" (OuterVolumeSpecName: "kube-api-access-pfngd") pod "b21e273d-207d-419a-a0e6-b9954be290df" (UID: "b21e273d-207d-419a-a0e6-b9954be290df"). InnerVolumeSpecName "kube-api-access-pfngd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:30:01.774073 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.774052    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21e273d-207d-419a-a0e6-b9954be290df-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b21e273d-207d-419a-a0e6-b9954be290df" (UID: "b21e273d-207d-419a-a0e6-b9954be290df"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:30:01.870351 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.870276    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b21e273d-207d-419a-a0e6-b9954be290df-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:30:01.870351 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.870312    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b21e273d-207d-419a-a0e6-b9954be290df-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:30:01.870351 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.870322    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfngd\" (UniqueName: \"kubernetes.io/projected/b21e273d-207d-419a-a0e6-b9954be290df-kube-api-access-pfngd\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:30:01.870351 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.870334    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b21e273d-207d-419a-a0e6-b9954be290df-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:30:01.870351 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:01.870343    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b21e273d-207d-419a-a0e6-b9954be290df-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:30:02.032251 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:02.032221    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"]
Jul 28 17:30:02.035865 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:02.035843    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-xcl6d"]
Jul 28 17:30:02.275497 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:02.275466    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:02.289166 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:02.289131    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b21e273d-207d-419a-a0e6-b9954be290df" path="/var/lib/kubelet/pods/b21e273d-207d-419a-a0e6-b9954be290df/volumes"
Jul 28 17:30:05.571052 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.570970    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:05.571584 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.571562    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"]
Jul 28 17:30:05.571827 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.571815    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b21e273d-207d-419a-a0e6-b9954be290df" containerName="storage-initializer"
Jul 28 17:30:05.571862 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.571830    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b21e273d-207d-419a-a0e6-b9954be290df" containerName="storage-initializer"
Jul 28 17:30:05.571862 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.571843    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b21e273d-207d-419a-a0e6-b9954be290df" containerName="kserve-container"
Jul 28 17:30:05.571862 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.571849    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b21e273d-207d-419a-a0e6-b9954be290df" containerName="kserve-container"
Jul 28 17:30:05.571983 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.571861    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b21e273d-207d-419a-a0e6-b9954be290df" containerName="kube-rbac-proxy"
Jul 28 17:30:05.571983 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.571870    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b21e273d-207d-419a-a0e6-b9954be290df" containerName="kube-rbac-proxy"
Jul 28 17:30:05.571983 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.571912    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b21e273d-207d-419a-a0e6-b9954be290df" containerName="kube-rbac-proxy"
Jul 28 17:30:05.571983 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.571919    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b21e273d-207d-419a-a0e6-b9954be290df" containerName="kserve-container"
Jul 28 17:30:05.574880 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.574863    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"
Jul 28 17:30:05.577826 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.577804    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 28 17:30:05.578074 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.577892    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:30:05.578186 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.577968    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 28 17:30:05.578337 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.578315    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:30:05.578430 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.578345    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 17:30:05.579406 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.579388    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:30:05.579517 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.579414    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:30:05.585836 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.585814    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"]
Jul 28 17:30:05.595278 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.595259    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc\" (UID: \"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"
Jul 28 17:30:05.595359 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.595289    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc\" (UID: \"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"
Jul 28 17:30:05.595400 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.595367    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x55x7\" (UniqueName: \"kubernetes.io/projected/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-kube-api-access-x55x7\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc\" (UID: \"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"
Jul 28 17:30:05.595435 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.595418    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc\" (UID: \"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"
Jul 28 17:30:05.595500 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.595480    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc\" (UID: \"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"
Jul 28 17:30:05.696527 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.696501    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc\" (UID: \"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"
Jul 28 17:30:05.696681 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.696540    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc\" (UID: \"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"
Jul 28 17:30:05.696681 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.696566    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc\" (UID: \"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"
Jul 28 17:30:05.696681 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.696581    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc\" (UID: \"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"
Jul 28 17:30:05.696681 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.696619    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x55x7\" (UniqueName: \"kubernetes.io/projected/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-kube-api-access-x55x7\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc\" (UID: \"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"
Jul 28 17:30:05.696681 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:30:05.696670    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-serving-cert: secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Jul 28 17:30:05.696925 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:30:05.696750    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-proxy-tls podName:c2d65a97-fcb5-488d-a4d4-5ee4765cadcc nodeName:}" failed. No retries permitted until 2026-07-28 17:30:06.196728577 +0000 UTC m=+1254.471345001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-proxy-tls") pod "isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc" (UID: "c2d65a97-fcb5-488d-a4d4-5ee4765cadcc") : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Jul 28 17:30:05.697050 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.697024    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc\" (UID: \"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"
Jul 28 17:30:05.697373 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.697350    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc\" (UID: \"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"
Jul 28 17:30:05.697467 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.697452    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc\" (UID: \"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"
Jul 28 17:30:05.705910 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.705881    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x55x7\" (UniqueName: \"kubernetes.io/projected/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-kube-api-access-x55x7\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc\" (UID: \"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"
Jul 28 17:30:05.794328 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:05.794293    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:30:06.200202 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:06.200150    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc\" (UID: \"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"
Jul 28 17:30:06.202511 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:06.202481    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc\" (UID: \"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"
Jul 28 17:30:06.485429 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:06.485340    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"
Jul 28 17:30:06.608174 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:06.608151    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"]
Jul 28 17:30:06.610583 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:30:06.610556    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d65a97_fcb5_488d_a4d4_5ee4765cadcc.slice/crio-f5c84f5afb3f5a2f729f5ffc5080ec7a3a9d5aa6c303b383576f08d9aed11211 WatchSource:0}: Error finding container f5c84f5afb3f5a2f729f5ffc5080ec7a3a9d5aa6c303b383576f08d9aed11211: Status 404 returned error can't find the container with id f5c84f5afb3f5a2f729f5ffc5080ec7a3a9d5aa6c303b383576f08d9aed11211
Jul 28 17:30:06.730465 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:06.730428    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc" event={"ID":"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc","Type":"ContainerStarted","Data":"3dc6c0558b1b017a5040c01b842f67e4f0d5dc503d1d64650b6dc7ac74e9caf7"}
Jul 28 17:30:06.730609 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:06.730512    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc" event={"ID":"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc","Type":"ContainerStarted","Data":"f5c84f5afb3f5a2f729f5ffc5080ec7a3a9d5aa6c303b383576f08d9aed11211"}
Jul 28 17:30:11.748258 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:11.748221    2581 generic.go:356] "Generic (PLEG): container finished" podID="c2d65a97-fcb5-488d-a4d4-5ee4765cadcc" containerID="3dc6c0558b1b017a5040c01b842f67e4f0d5dc503d1d64650b6dc7ac74e9caf7" exitCode=0
Jul 28 17:30:11.748654 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:11.748289    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc" event={"ID":"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc","Type":"ContainerDied","Data":"3dc6c0558b1b017a5040c01b842f67e4f0d5dc503d1d64650b6dc7ac74e9caf7"}
Jul 28 17:30:12.754484 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:12.754448    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc" event={"ID":"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc","Type":"ContainerStarted","Data":"39e2dea4e533e7fd1566515f91ac29db8dc451dc8d4c2f7f774d78e5ab96050c"}
Jul 28 17:30:12.754887 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:12.754534    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc" event={"ID":"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc","Type":"ContainerStarted","Data":"c9a67a2e2eedeeecae14eb5418af17f35f976e5bc990baa9ad6d3cde97f5dcc4"}
Jul 28 17:30:12.754887 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:12.754754    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"
Jul 28 17:30:12.755015 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:12.754933    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"
Jul 28 17:30:12.779034 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:12.778985    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc" podStartSLOduration=2.919593204 podStartE2EDuration="7.778970671s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.859377467s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:30:12.776648331 +0000 UTC m=+1261.051264766" watchObservedRunningTime="2026-07-28 17:30:12.778970671 +0000 UTC m=+1261.053587103"
Jul 28 17:30:18.765423 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:30:18.765390    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"
Jul 28 17:31:48.290659 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:31:48.290581    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"
Jul 28 17:31:59.328110 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:31:59.328062    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc" podUID="c2d65a97-fcb5-488d-a4d4-5ee4765cadcc" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.23:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 17:32:02.124575 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:02.124527    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"]
Jul 28 17:32:02.125043 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:02.124984    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc" podUID="c2d65a97-fcb5-488d-a4d4-5ee4765cadcc" containerName="kserve-container" containerID="cri-o://c9a67a2e2eedeeecae14eb5418af17f35f976e5bc990baa9ad6d3cde97f5dcc4" gracePeriod=30
Jul 28 17:32:02.126025 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:02.125432    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc" podUID="c2d65a97-fcb5-488d-a4d4-5ee4765cadcc" containerName="kube-rbac-proxy" containerID="cri-o://39e2dea4e533e7fd1566515f91ac29db8dc451dc8d4c2f7f774d78e5ab96050c" gracePeriod=30
Jul 28 17:32:02.170900 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:02.170878    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:03.137393 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:03.137354    2581 generic.go:356] "Generic (PLEG): container finished" podID="c2d65a97-fcb5-488d-a4d4-5ee4765cadcc" containerID="39e2dea4e533e7fd1566515f91ac29db8dc451dc8d4c2f7f774d78e5ab96050c" exitCode=2
Jul 28 17:32:03.137761 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:03.137428    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc" event={"ID":"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc","Type":"ContainerDied","Data":"39e2dea4e533e7fd1566515f91ac29db8dc451dc8d4c2f7f774d78e5ab96050c"}
Jul 28 17:32:03.467721 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:03.467701    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"
Jul 28 17:32:03.512749 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:03.512724    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-kserve-provision-location\") pod \"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc\" (UID: \"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc\") "
Jul 28 17:32:03.512864 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:03.512758    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-kube-rbac-proxy-sar-config\") pod \"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc\" (UID: \"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc\") "
Jul 28 17:32:03.512864 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:03.512789    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-proxy-tls\") pod \"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc\" (UID: \"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc\") "
Jul 28 17:32:03.512864 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:03.512811    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-cabundle-cert\") pod \"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc\" (UID: \"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc\") "
Jul 28 17:32:03.513043 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:03.512864    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x55x7\" (UniqueName: \"kubernetes.io/projected/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-kube-api-access-x55x7\") pod \"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc\" (UID: \"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc\") "
Jul 28 17:32:03.513191 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:03.513074    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c2d65a97-fcb5-488d-a4d4-5ee4765cadcc" (UID: "c2d65a97-fcb5-488d-a4d4-5ee4765cadcc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:32:03.513309 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:03.513216    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c2d65a97-fcb5-488d-a4d4-5ee4765cadcc" (UID: "c2d65a97-fcb5-488d-a4d4-5ee4765cadcc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:32:03.514751 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:03.514727    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c2d65a97-fcb5-488d-a4d4-5ee4765cadcc" (UID: "c2d65a97-fcb5-488d-a4d4-5ee4765cadcc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:32:03.514851 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:03.514831    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-kube-api-access-x55x7" (OuterVolumeSpecName: "kube-api-access-x55x7") pod "c2d65a97-fcb5-488d-a4d4-5ee4765cadcc" (UID: "c2d65a97-fcb5-488d-a4d4-5ee4765cadcc"). InnerVolumeSpecName "kube-api-access-x55x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:32:03.516981 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:03.516956    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c2d65a97-fcb5-488d-a4d4-5ee4765cadcc" (UID: "c2d65a97-fcb5-488d-a4d4-5ee4765cadcc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:32:03.613847 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:03.613811    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x55x7\" (UniqueName: \"kubernetes.io/projected/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-kube-api-access-x55x7\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:32:03.613847 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:03.613842    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:32:03.613847 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:03.613852    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:32:03.614045 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:03.613861    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:32:03.614045 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:03.613869    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:32:04.143116 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:04.143079    2581 generic.go:356] "Generic (PLEG): container finished" podID="c2d65a97-fcb5-488d-a4d4-5ee4765cadcc" containerID="c9a67a2e2eedeeecae14eb5418af17f35f976e5bc990baa9ad6d3cde97f5dcc4" exitCode=0
Jul 28 17:32:04.143564 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:04.143121    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc" event={"ID":"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc","Type":"ContainerDied","Data":"c9a67a2e2eedeeecae14eb5418af17f35f976e5bc990baa9ad6d3cde97f5dcc4"}
Jul 28 17:32:04.143564 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:04.143135    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"
Jul 28 17:32:04.143564 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:04.143180    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc" event={"ID":"c2d65a97-fcb5-488d-a4d4-5ee4765cadcc","Type":"ContainerDied","Data":"f5c84f5afb3f5a2f729f5ffc5080ec7a3a9d5aa6c303b383576f08d9aed11211"}
Jul 28 17:32:04.143564 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:04.143219    2581 scope.go:117] "RemoveContainer" containerID="39e2dea4e533e7fd1566515f91ac29db8dc451dc8d4c2f7f774d78e5ab96050c"
Jul 28 17:32:04.151259 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:04.151245    2581 scope.go:117] "RemoveContainer" containerID="c9a67a2e2eedeeecae14eb5418af17f35f976e5bc990baa9ad6d3cde97f5dcc4"
Jul 28 17:32:04.158257 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:04.158243    2581 scope.go:117] "RemoveContainer" containerID="3dc6c0558b1b017a5040c01b842f67e4f0d5dc503d1d64650b6dc7ac74e9caf7"
Jul 28 17:32:04.165046 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:04.165026    2581 scope.go:117] "RemoveContainer" containerID="39e2dea4e533e7fd1566515f91ac29db8dc451dc8d4c2f7f774d78e5ab96050c"
Jul 28 17:32:04.165308 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:32:04.165289    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39e2dea4e533e7fd1566515f91ac29db8dc451dc8d4c2f7f774d78e5ab96050c\": container with ID starting with 39e2dea4e533e7fd1566515f91ac29db8dc451dc8d4c2f7f774d78e5ab96050c not found: ID does not exist" containerID="39e2dea4e533e7fd1566515f91ac29db8dc451dc8d4c2f7f774d78e5ab96050c"
Jul 28 17:32:04.165355 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:04.165317    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39e2dea4e533e7fd1566515f91ac29db8dc451dc8d4c2f7f774d78e5ab96050c"} err="failed to get container status \"39e2dea4e533e7fd1566515f91ac29db8dc451dc8d4c2f7f774d78e5ab96050c\": rpc error: code = NotFound desc = could not find container \"39e2dea4e533e7fd1566515f91ac29db8dc451dc8d4c2f7f774d78e5ab96050c\": container with ID starting with 39e2dea4e533e7fd1566515f91ac29db8dc451dc8d4c2f7f774d78e5ab96050c not found: ID does not exist"
Jul 28 17:32:04.165355 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:04.165334    2581 scope.go:117] "RemoveContainer" containerID="c9a67a2e2eedeeecae14eb5418af17f35f976e5bc990baa9ad6d3cde97f5dcc4"
Jul 28 17:32:04.165598 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:32:04.165579    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9a67a2e2eedeeecae14eb5418af17f35f976e5bc990baa9ad6d3cde97f5dcc4\": container with ID starting with c9a67a2e2eedeeecae14eb5418af17f35f976e5bc990baa9ad6d3cde97f5dcc4 not found: ID does not exist" containerID="c9a67a2e2eedeeecae14eb5418af17f35f976e5bc990baa9ad6d3cde97f5dcc4"
Jul 28 17:32:04.165663 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:04.165604    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a67a2e2eedeeecae14eb5418af17f35f976e5bc990baa9ad6d3cde97f5dcc4"} err="failed to get container status \"c9a67a2e2eedeeecae14eb5418af17f35f976e5bc990baa9ad6d3cde97f5dcc4\": rpc error: code = NotFound desc = could not find container \"c9a67a2e2eedeeecae14eb5418af17f35f976e5bc990baa9ad6d3cde97f5dcc4\": container with ID starting with c9a67a2e2eedeeecae14eb5418af17f35f976e5bc990baa9ad6d3cde97f5dcc4 not found: ID does not exist"
Jul 28 17:32:04.165663 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:04.165621    2581 scope.go:117] "RemoveContainer" containerID="3dc6c0558b1b017a5040c01b842f67e4f0d5dc503d1d64650b6dc7ac74e9caf7"
Jul 28 17:32:04.165871 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:32:04.165852    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dc6c0558b1b017a5040c01b842f67e4f0d5dc503d1d64650b6dc7ac74e9caf7\": container with ID starting with 3dc6c0558b1b017a5040c01b842f67e4f0d5dc503d1d64650b6dc7ac74e9caf7 not found: ID does not exist" containerID="3dc6c0558b1b017a5040c01b842f67e4f0d5dc503d1d64650b6dc7ac74e9caf7"
Jul 28 17:32:04.165925 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:04.165879    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dc6c0558b1b017a5040c01b842f67e4f0d5dc503d1d64650b6dc7ac74e9caf7"} err="failed to get container status \"3dc6c0558b1b017a5040c01b842f67e4f0d5dc503d1d64650b6dc7ac74e9caf7\": rpc error: code = NotFound desc = could not find container \"3dc6c0558b1b017a5040c01b842f67e4f0d5dc503d1d64650b6dc7ac74e9caf7\": container with ID starting with 3dc6c0558b1b017a5040c01b842f67e4f0d5dc503d1d64650b6dc7ac74e9caf7 not found: ID does not exist"
Jul 28 17:32:04.166142 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:04.166126    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"]
Jul 28 17:32:04.170965 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:04.170928    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-tqmvc"]
Jul 28 17:32:04.289085 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:04.289060    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2d65a97-fcb5-488d-a4d4-5ee4765cadcc" path="/var/lib/kubelet/pods/c2d65a97-fcb5-488d-a4d4-5ee4765cadcc/volumes"
Jul 28 17:32:07.188453 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.188414    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:07.256030 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.256001    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"]
Jul 28 17:32:07.256276 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.256264    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2d65a97-fcb5-488d-a4d4-5ee4765cadcc" containerName="kserve-container"
Jul 28 17:32:07.256320 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.256277    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2d65a97-fcb5-488d-a4d4-5ee4765cadcc" containerName="kserve-container"
Jul 28 17:32:07.256320 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.256287    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2d65a97-fcb5-488d-a4d4-5ee4765cadcc" containerName="storage-initializer"
Jul 28 17:32:07.256320 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.256293    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2d65a97-fcb5-488d-a4d4-5ee4765cadcc" containerName="storage-initializer"
Jul 28 17:32:07.256320 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.256299    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2d65a97-fcb5-488d-a4d4-5ee4765cadcc" containerName="kube-rbac-proxy"
Jul 28 17:32:07.256320 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.256305    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2d65a97-fcb5-488d-a4d4-5ee4765cadcc" containerName="kube-rbac-proxy"
Jul 28 17:32:07.256479 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.256364    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2d65a97-fcb5-488d-a4d4-5ee4765cadcc" containerName="kube-rbac-proxy"
Jul 28 17:32:07.256479 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.256375    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2d65a97-fcb5-488d-a4d4-5ee4765cadcc" containerName="kserve-container"
Jul 28 17:32:07.259317 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.259301    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"
Jul 28 17:32:07.263676 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.263642    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 17:32:07.263809 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.263728    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:32:07.263809 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.263736    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:32:07.263809 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.263731    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 28 17:32:07.263809 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.263744    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:32:07.263809 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.263739    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:32:07.264063 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.263873    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 28 17:32:07.273750 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.273733    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"]
Jul 28 17:32:07.293658 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.293634    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:32:07.337791 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.337765    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a24144f-077a-4e1c-95d8-4d66efe56323-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb\" (UID: \"6a24144f-077a-4e1c-95d8-4d66efe56323\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"
Jul 28 17:32:07.337891 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.337801    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a24144f-077a-4e1c-95d8-4d66efe56323-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb\" (UID: \"6a24144f-077a-4e1c-95d8-4d66efe56323\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"
Jul 28 17:32:07.337891 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.337834    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a24144f-077a-4e1c-95d8-4d66efe56323-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb\" (UID: \"6a24144f-077a-4e1c-95d8-4d66efe56323\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"
Jul 28 17:32:07.337992 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.337926    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrjpm\" (UniqueName: \"kubernetes.io/projected/6a24144f-077a-4e1c-95d8-4d66efe56323-kube-api-access-zrjpm\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb\" (UID: \"6a24144f-077a-4e1c-95d8-4d66efe56323\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"
Jul 28 17:32:07.338027 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.338012    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a24144f-077a-4e1c-95d8-4d66efe56323-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb\" (UID: \"6a24144f-077a-4e1c-95d8-4d66efe56323\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"
Jul 28 17:32:07.439180 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.439120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a24144f-077a-4e1c-95d8-4d66efe56323-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb\" (UID: \"6a24144f-077a-4e1c-95d8-4d66efe56323\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"
Jul 28 17:32:07.439180 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.439169    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrjpm\" (UniqueName: \"kubernetes.io/projected/6a24144f-077a-4e1c-95d8-4d66efe56323-kube-api-access-zrjpm\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb\" (UID: \"6a24144f-077a-4e1c-95d8-4d66efe56323\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"
Jul 28 17:32:07.439318 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.439211    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a24144f-077a-4e1c-95d8-4d66efe56323-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb\" (UID: \"6a24144f-077a-4e1c-95d8-4d66efe56323\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"
Jul 28 17:32:07.439318 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.439230    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a24144f-077a-4e1c-95d8-4d66efe56323-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb\" (UID: \"6a24144f-077a-4e1c-95d8-4d66efe56323\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"
Jul 28 17:32:07.439318 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.439248    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a24144f-077a-4e1c-95d8-4d66efe56323-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb\" (UID: \"6a24144f-077a-4e1c-95d8-4d66efe56323\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"
Jul 28 17:32:07.439573 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.439552    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a24144f-077a-4e1c-95d8-4d66efe56323-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb\" (UID: \"6a24144f-077a-4e1c-95d8-4d66efe56323\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"
Jul 28 17:32:07.439826 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.439796    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a24144f-077a-4e1c-95d8-4d66efe56323-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb\" (UID: \"6a24144f-077a-4e1c-95d8-4d66efe56323\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"
Jul 28 17:32:07.439904 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.439862    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a24144f-077a-4e1c-95d8-4d66efe56323-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb\" (UID: \"6a24144f-077a-4e1c-95d8-4d66efe56323\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"
Jul 28 17:32:07.441667 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.441644    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a24144f-077a-4e1c-95d8-4d66efe56323-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb\" (UID: \"6a24144f-077a-4e1c-95d8-4d66efe56323\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"
Jul 28 17:32:07.447920 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.447901    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrjpm\" (UniqueName: \"kubernetes.io/projected/6a24144f-077a-4e1c-95d8-4d66efe56323-kube-api-access-zrjpm\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb\" (UID: \"6a24144f-077a-4e1c-95d8-4d66efe56323\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"
Jul 28 17:32:07.569310 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.569275    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"
Jul 28 17:32:07.699308 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:07.699282    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"]
Jul 28 17:32:07.701637 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:32:07.701611    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a24144f_077a_4e1c_95d8_4d66efe56323.slice/crio-fbe900c41b7831bab15105d70463cd4350430215d9a5125b4e1b64061346864f WatchSource:0}: Error finding container fbe900c41b7831bab15105d70463cd4350430215d9a5125b4e1b64061346864f: Status 404 returned error can't find the container with id fbe900c41b7831bab15105d70463cd4350430215d9a5125b4e1b64061346864f
Jul 28 17:32:08.159807 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:08.159763    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb" event={"ID":"6a24144f-077a-4e1c-95d8-4d66efe56323","Type":"ContainerStarted","Data":"1070042b3879c7c7128e43f732acdea39ca9c98c580eb85d1216400997390080"}
Jul 28 17:32:08.159984 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:08.159837    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb" event={"ID":"6a24144f-077a-4e1c-95d8-4d66efe56323","Type":"ContainerStarted","Data":"fbe900c41b7831bab15105d70463cd4350430215d9a5125b4e1b64061346864f"}
Jul 28 17:32:28.231231 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:28.231195    2581 generic.go:356] "Generic (PLEG): container finished" podID="6a24144f-077a-4e1c-95d8-4d66efe56323" containerID="1070042b3879c7c7128e43f732acdea39ca9c98c580eb85d1216400997390080" exitCode=0
Jul 28 17:32:28.231624 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:28.231268    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb" event={"ID":"6a24144f-077a-4e1c-95d8-4d66efe56323","Type":"ContainerDied","Data":"1070042b3879c7c7128e43f732acdea39ca9c98c580eb85d1216400997390080"}
Jul 28 17:32:29.236909 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:29.236873    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb" event={"ID":"6a24144f-077a-4e1c-95d8-4d66efe56323","Type":"ContainerStarted","Data":"68c37d655e793ad255f8754f2d7192dc3e7ca80325e8d3859670c33aa3461c0e"}
Jul 28 17:32:29.237361 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:29.236933    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb" event={"ID":"6a24144f-077a-4e1c-95d8-4d66efe56323","Type":"ContainerStarted","Data":"74cc0cf225af7eb258da9c4e1d51b188e9c59a7b5b7566ab197be7467dd9340a"}
Jul 28 17:32:29.237361 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:29.237272    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"
Jul 28 17:32:29.237361 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:29.237317    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"
Jul 28 17:32:29.262598 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:29.262552    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb" podStartSLOduration=2.14461192 podStartE2EDuration="22.262534495s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.117922575s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:32:29.260060872 +0000 UTC m=+1397.534677296" watchObservedRunningTime="2026-07-28 17:32:29.262534495 +0000 UTC m=+1397.537150928"
Jul 28 17:32:35.247250 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:32:35.247218    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"
Jul 28 17:34:03.289541 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:03.289503    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"
Jul 28 17:34:12.256580 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:12.256552    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 17:34:12.257341 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:12.257317    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 17:34:13.437237 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:13.437205    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"]
Jul 28 17:34:13.437675 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:13.437508    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb" podUID="6a24144f-077a-4e1c-95d8-4d66efe56323" containerName="kserve-container" containerID="cri-o://74cc0cf225af7eb258da9c4e1d51b188e9c59a7b5b7566ab197be7467dd9340a" gracePeriod=30
Jul 28 17:34:13.437675 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:13.437587    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb" podUID="6a24144f-077a-4e1c-95d8-4d66efe56323" containerName="kube-rbac-proxy" containerID="cri-o://68c37d655e793ad255f8754f2d7192dc3e7ca80325e8d3859670c33aa3461c0e" gracePeriod=30
Jul 28 17:34:13.581528 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:13.581460    2581 generic.go:356] "Generic (PLEG): container finished" podID="6a24144f-077a-4e1c-95d8-4d66efe56323" containerID="68c37d655e793ad255f8754f2d7192dc3e7ca80325e8d3859670c33aa3461c0e" exitCode=2
Jul 28 17:34:13.581528 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:13.581495    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb" event={"ID":"6a24144f-077a-4e1c-95d8-4d66efe56323","Type":"ContainerDied","Data":"68c37d655e793ad255f8754f2d7192dc3e7ca80325e8d3859670c33aa3461c0e"}
Jul 28 17:34:14.242076 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:14.242041    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:14.771025 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:14.771002    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"
Jul 28 17:34:14.832493 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:14.832433    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a24144f-077a-4e1c-95d8-4d66efe56323-kube-rbac-proxy-sar-config\") pod \"6a24144f-077a-4e1c-95d8-4d66efe56323\" (UID: \"6a24144f-077a-4e1c-95d8-4d66efe56323\") "
Jul 28 17:34:14.832612 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:14.832495    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrjpm\" (UniqueName: \"kubernetes.io/projected/6a24144f-077a-4e1c-95d8-4d66efe56323-kube-api-access-zrjpm\") pod \"6a24144f-077a-4e1c-95d8-4d66efe56323\" (UID: \"6a24144f-077a-4e1c-95d8-4d66efe56323\") "
Jul 28 17:34:14.832612 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:14.832518    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a24144f-077a-4e1c-95d8-4d66efe56323-cabundle-cert\") pod \"6a24144f-077a-4e1c-95d8-4d66efe56323\" (UID: \"6a24144f-077a-4e1c-95d8-4d66efe56323\") "
Jul 28 17:34:14.832612 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:14.832583    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a24144f-077a-4e1c-95d8-4d66efe56323-kserve-provision-location\") pod \"6a24144f-077a-4e1c-95d8-4d66efe56323\" (UID: \"6a24144f-077a-4e1c-95d8-4d66efe56323\") "
Jul 28 17:34:14.832747 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:14.832629    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a24144f-077a-4e1c-95d8-4d66efe56323-proxy-tls\") pod \"6a24144f-077a-4e1c-95d8-4d66efe56323\" (UID: \"6a24144f-077a-4e1c-95d8-4d66efe56323\") "
Jul 28 17:34:14.832958 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:14.832909    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a24144f-077a-4e1c-95d8-4d66efe56323-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6a24144f-077a-4e1c-95d8-4d66efe56323" (UID: "6a24144f-077a-4e1c-95d8-4d66efe56323"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:34:14.832958 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:14.832915    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a24144f-077a-4e1c-95d8-4d66efe56323-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6a24144f-077a-4e1c-95d8-4d66efe56323" (UID: "6a24144f-077a-4e1c-95d8-4d66efe56323"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:34:14.834489 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:14.834466    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a24144f-077a-4e1c-95d8-4d66efe56323-kube-api-access-zrjpm" (OuterVolumeSpecName: "kube-api-access-zrjpm") pod "6a24144f-077a-4e1c-95d8-4d66efe56323" (UID: "6a24144f-077a-4e1c-95d8-4d66efe56323"). InnerVolumeSpecName "kube-api-access-zrjpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:34:14.834582 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:14.834542    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a24144f-077a-4e1c-95d8-4d66efe56323-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a24144f-077a-4e1c-95d8-4d66efe56323" (UID: "6a24144f-077a-4e1c-95d8-4d66efe56323"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:34:14.837096 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:14.837067    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a24144f-077a-4e1c-95d8-4d66efe56323-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6a24144f-077a-4e1c-95d8-4d66efe56323" (UID: "6a24144f-077a-4e1c-95d8-4d66efe56323"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:34:14.934012 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:14.933989    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrjpm\" (UniqueName: \"kubernetes.io/projected/6a24144f-077a-4e1c-95d8-4d66efe56323-kube-api-access-zrjpm\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:34:14.934012 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:14.934010    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a24144f-077a-4e1c-95d8-4d66efe56323-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:34:14.934154 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:14.934021    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a24144f-077a-4e1c-95d8-4d66efe56323-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:34:14.934154 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:14.934030    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a24144f-077a-4e1c-95d8-4d66efe56323-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:34:14.934154 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:14.934040    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a24144f-077a-4e1c-95d8-4d66efe56323-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:34:15.589987 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:15.589934    2581 generic.go:356] "Generic (PLEG): container finished" podID="6a24144f-077a-4e1c-95d8-4d66efe56323" containerID="74cc0cf225af7eb258da9c4e1d51b188e9c59a7b5b7566ab197be7467dd9340a" exitCode=0
Jul 28 17:34:15.590148 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:15.590011    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"
Jul 28 17:34:15.590148 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:15.590021    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb" event={"ID":"6a24144f-077a-4e1c-95d8-4d66efe56323","Type":"ContainerDied","Data":"74cc0cf225af7eb258da9c4e1d51b188e9c59a7b5b7566ab197be7467dd9340a"}
Jul 28 17:34:15.590148 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:15.590082    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb" event={"ID":"6a24144f-077a-4e1c-95d8-4d66efe56323","Type":"ContainerDied","Data":"fbe900c41b7831bab15105d70463cd4350430215d9a5125b4e1b64061346864f"}
Jul 28 17:34:15.590148 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:15.590108    2581 scope.go:117] "RemoveContainer" containerID="68c37d655e793ad255f8754f2d7192dc3e7ca80325e8d3859670c33aa3461c0e"
Jul 28 17:34:15.597929 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:15.597906    2581 scope.go:117] "RemoveContainer" containerID="74cc0cf225af7eb258da9c4e1d51b188e9c59a7b5b7566ab197be7467dd9340a"
Jul 28 17:34:15.604807 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:15.604792    2581 scope.go:117] "RemoveContainer" containerID="1070042b3879c7c7128e43f732acdea39ca9c98c580eb85d1216400997390080"
Jul 28 17:34:15.611232 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:15.611218    2581 scope.go:117] "RemoveContainer" containerID="68c37d655e793ad255f8754f2d7192dc3e7ca80325e8d3859670c33aa3461c0e"
Jul 28 17:34:15.611451 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:34:15.611409    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68c37d655e793ad255f8754f2d7192dc3e7ca80325e8d3859670c33aa3461c0e\": container with ID starting with 68c37d655e793ad255f8754f2d7192dc3e7ca80325e8d3859670c33aa3461c0e not found: ID does not exist" containerID="68c37d655e793ad255f8754f2d7192dc3e7ca80325e8d3859670c33aa3461c0e"
Jul 28 17:34:15.611451 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:15.611433    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68c37d655e793ad255f8754f2d7192dc3e7ca80325e8d3859670c33aa3461c0e"} err="failed to get container status \"68c37d655e793ad255f8754f2d7192dc3e7ca80325e8d3859670c33aa3461c0e\": rpc error: code = NotFound desc = could not find container \"68c37d655e793ad255f8754f2d7192dc3e7ca80325e8d3859670c33aa3461c0e\": container with ID starting with 68c37d655e793ad255f8754f2d7192dc3e7ca80325e8d3859670c33aa3461c0e not found: ID does not exist"
Jul 28 17:34:15.611626 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:15.611457    2581 scope.go:117] "RemoveContainer" containerID="74cc0cf225af7eb258da9c4e1d51b188e9c59a7b5b7566ab197be7467dd9340a"
Jul 28 17:34:15.611828 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:34:15.611801    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74cc0cf225af7eb258da9c4e1d51b188e9c59a7b5b7566ab197be7467dd9340a\": container with ID starting with 74cc0cf225af7eb258da9c4e1d51b188e9c59a7b5b7566ab197be7467dd9340a not found: ID does not exist" containerID="74cc0cf225af7eb258da9c4e1d51b188e9c59a7b5b7566ab197be7467dd9340a"
Jul 28 17:34:15.611917 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:15.611836    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74cc0cf225af7eb258da9c4e1d51b188e9c59a7b5b7566ab197be7467dd9340a"} err="failed to get container status \"74cc0cf225af7eb258da9c4e1d51b188e9c59a7b5b7566ab197be7467dd9340a\": rpc error: code = NotFound desc = could not find container \"74cc0cf225af7eb258da9c4e1d51b188e9c59a7b5b7566ab197be7467dd9340a\": container with ID starting with 74cc0cf225af7eb258da9c4e1d51b188e9c59a7b5b7566ab197be7467dd9340a not found: ID does not exist"
Jul 28 17:34:15.611917 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:15.611856    2581 scope.go:117] "RemoveContainer" containerID="1070042b3879c7c7128e43f732acdea39ca9c98c580eb85d1216400997390080"
Jul 28 17:34:15.612200 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:34:15.612140    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1070042b3879c7c7128e43f732acdea39ca9c98c580eb85d1216400997390080\": container with ID starting with 1070042b3879c7c7128e43f732acdea39ca9c98c580eb85d1216400997390080 not found: ID does not exist" containerID="1070042b3879c7c7128e43f732acdea39ca9c98c580eb85d1216400997390080"
Jul 28 17:34:15.612200 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:15.612170    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1070042b3879c7c7128e43f732acdea39ca9c98c580eb85d1216400997390080"} err="failed to get container status \"1070042b3879c7c7128e43f732acdea39ca9c98c580eb85d1216400997390080\": rpc error: code = NotFound desc = could not find container \"1070042b3879c7c7128e43f732acdea39ca9c98c580eb85d1216400997390080\": container with ID starting with 1070042b3879c7c7128e43f732acdea39ca9c98c580eb85d1216400997390080 not found: ID does not exist"
Jul 28 17:34:15.612774 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:15.612756    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"]
Jul 28 17:34:15.617464 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:15.617443    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-hdgvb"]
Jul 28 17:34:16.289342 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:16.289309    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a24144f-077a-4e1c-95d8-4d66efe56323" path="/var/lib/kubelet/pods/6a24144f-077a-4e1c-95d8-4d66efe56323/volumes"
Jul 28 17:34:18.485804 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.485774    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:18.547837 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.547805    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"]
Jul 28 17:34:18.548084 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.548072    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a24144f-077a-4e1c-95d8-4d66efe56323" containerName="kube-rbac-proxy"
Jul 28 17:34:18.548128 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.548086    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a24144f-077a-4e1c-95d8-4d66efe56323" containerName="kube-rbac-proxy"
Jul 28 17:34:18.548128 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.548104    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a24144f-077a-4e1c-95d8-4d66efe56323" containerName="storage-initializer"
Jul 28 17:34:18.548128 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.548110    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a24144f-077a-4e1c-95d8-4d66efe56323" containerName="storage-initializer"
Jul 28 17:34:18.548128 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.548120    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a24144f-077a-4e1c-95d8-4d66efe56323" containerName="kserve-container"
Jul 28 17:34:18.548128 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.548126    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a24144f-077a-4e1c-95d8-4d66efe56323" containerName="kserve-container"
Jul 28 17:34:18.548284 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.548171    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a24144f-077a-4e1c-95d8-4d66efe56323" containerName="kube-rbac-proxy"
Jul 28 17:34:18.548284 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.548182    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a24144f-077a-4e1c-95d8-4d66efe56323" containerName="kserve-container"
Jul 28 17:34:18.552651 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.552633    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"
Jul 28 17:34:18.555402 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.555373    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 28 17:34:18.555533 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.555513    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:34:18.555533 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.555529    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:34:18.555683 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.555574    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:34:18.555683 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.555659    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 28 17:34:18.555683 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.555667    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:34:18.556687 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.556671    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 17:34:18.561903 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.561873    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"]
Jul 28 17:34:18.661569 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.661537    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x67hs\" (UniqueName: \"kubernetes.io/projected/083e2a6e-ffc7-407b-9b75-b26340519901-kube-api-access-x67hs\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv\" (UID: \"083e2a6e-ffc7-407b-9b75-b26340519901\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"
Jul 28 17:34:18.661726 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.661579    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/083e2a6e-ffc7-407b-9b75-b26340519901-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv\" (UID: \"083e2a6e-ffc7-407b-9b75-b26340519901\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"
Jul 28 17:34:18.661726 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.661608    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/083e2a6e-ffc7-407b-9b75-b26340519901-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv\" (UID: \"083e2a6e-ffc7-407b-9b75-b26340519901\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"
Jul 28 17:34:18.661726 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.661681    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/083e2a6e-ffc7-407b-9b75-b26340519901-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv\" (UID: \"083e2a6e-ffc7-407b-9b75-b26340519901\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"
Jul 28 17:34:18.661726 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.661718    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/083e2a6e-ffc7-407b-9b75-b26340519901-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv\" (UID: \"083e2a6e-ffc7-407b-9b75-b26340519901\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"
Jul 28 17:34:18.762366 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.762274    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/083e2a6e-ffc7-407b-9b75-b26340519901-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv\" (UID: \"083e2a6e-ffc7-407b-9b75-b26340519901\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"
Jul 28 17:34:18.762366 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.762316    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/083e2a6e-ffc7-407b-9b75-b26340519901-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv\" (UID: \"083e2a6e-ffc7-407b-9b75-b26340519901\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"
Jul 28 17:34:18.762366 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.762359    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/083e2a6e-ffc7-407b-9b75-b26340519901-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv\" (UID: \"083e2a6e-ffc7-407b-9b75-b26340519901\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"
Jul 28 17:34:18.762642 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.762382    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/083e2a6e-ffc7-407b-9b75-b26340519901-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv\" (UID: \"083e2a6e-ffc7-407b-9b75-b26340519901\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"
Jul 28 17:34:18.762642 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.762442    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x67hs\" (UniqueName: \"kubernetes.io/projected/083e2a6e-ffc7-407b-9b75-b26340519901-kube-api-access-x67hs\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv\" (UID: \"083e2a6e-ffc7-407b-9b75-b26340519901\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"
Jul 28 17:34:18.762642 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:34:18.762514    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-serving-cert: secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Jul 28 17:34:18.762642 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:34:18.762598    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/083e2a6e-ffc7-407b-9b75-b26340519901-proxy-tls podName:083e2a6e-ffc7-407b-9b75-b26340519901 nodeName:}" failed. No retries permitted until 2026-07-28 17:34:19.262575975 +0000 UTC m=+1507.537192389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/083e2a6e-ffc7-407b-9b75-b26340519901-proxy-tls") pod "isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv" (UID: "083e2a6e-ffc7-407b-9b75-b26340519901") : secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Jul 28 17:34:18.762834 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.762699    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/083e2a6e-ffc7-407b-9b75-b26340519901-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv\" (UID: \"083e2a6e-ffc7-407b-9b75-b26340519901\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"
Jul 28 17:34:18.763076 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.763059    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/083e2a6e-ffc7-407b-9b75-b26340519901-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv\" (UID: \"083e2a6e-ffc7-407b-9b75-b26340519901\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"
Jul 28 17:34:18.763153 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.763133    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/083e2a6e-ffc7-407b-9b75-b26340519901-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv\" (UID: \"083e2a6e-ffc7-407b-9b75-b26340519901\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"
Jul 28 17:34:18.773268 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:18.773239    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x67hs\" (UniqueName: \"kubernetes.io/projected/083e2a6e-ffc7-407b-9b75-b26340519901-kube-api-access-x67hs\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv\" (UID: \"083e2a6e-ffc7-407b-9b75-b26340519901\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"
Jul 28 17:34:19.155242 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:19.155194    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:19.267339 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:19.267297    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/083e2a6e-ffc7-407b-9b75-b26340519901-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv\" (UID: \"083e2a6e-ffc7-407b-9b75-b26340519901\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"
Jul 28 17:34:19.269718 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:19.269697    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/083e2a6e-ffc7-407b-9b75-b26340519901-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv\" (UID: \"083e2a6e-ffc7-407b-9b75-b26340519901\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"
Jul 28 17:34:19.463295 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:19.463217    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"
Jul 28 17:34:19.589656 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:19.589627    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"]
Jul 28 17:34:19.593643 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:34:19.593618    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083e2a6e_ffc7_407b_9b75_b26340519901.slice/crio-c6711565f01e4c1415f553b498260090fbeb4f48a5ee069e922585cac920de95 WatchSource:0}: Error finding container c6711565f01e4c1415f553b498260090fbeb4f48a5ee069e922585cac920de95: Status 404 returned error can't find the container with id c6711565f01e4c1415f553b498260090fbeb4f48a5ee069e922585cac920de95
Jul 28 17:34:19.611573 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:19.611544    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv" event={"ID":"083e2a6e-ffc7-407b-9b75-b26340519901","Type":"ContainerStarted","Data":"c6711565f01e4c1415f553b498260090fbeb4f48a5ee069e922585cac920de95"}
Jul 28 17:34:20.616645 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:20.616612    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv" event={"ID":"083e2a6e-ffc7-407b-9b75-b26340519901","Type":"ContainerStarted","Data":"038b2269749576dd1cb78609927c4c27bf8faa2304b9b837278f6947938becb8"}
Jul 28 17:34:23.407634 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:23.407599    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:34:27.641319 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:27.641286    2581 generic.go:356] "Generic (PLEG): container finished" podID="083e2a6e-ffc7-407b-9b75-b26340519901" containerID="038b2269749576dd1cb78609927c4c27bf8faa2304b9b837278f6947938becb8" exitCode=0
Jul 28 17:34:27.641694 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:27.641359    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv" event={"ID":"083e2a6e-ffc7-407b-9b75-b26340519901","Type":"ContainerDied","Data":"038b2269749576dd1cb78609927c4c27bf8faa2304b9b837278f6947938becb8"}
Jul 28 17:34:28.646374 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:28.646334    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv" event={"ID":"083e2a6e-ffc7-407b-9b75-b26340519901","Type":"ContainerStarted","Data":"bba921d6597559828a38c6fbb92680e437f5ad23221d3d65185a09d7cc6b88c0"}
Jul 28 17:34:28.646847 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:28.646437    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv" event={"ID":"083e2a6e-ffc7-407b-9b75-b26340519901","Type":"ContainerStarted","Data":"8881c94fdee0fdd296ef2b8636114857a933277222ddb5dcf6ce48ffcf5071e6"}
Jul 28 17:34:28.646847 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:28.646657    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"
Jul 28 17:34:28.671657 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:28.671610    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv" podStartSLOduration=3.199590783 podStartE2EDuration="10.671597303s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.47200652s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:34:28.670074348 +0000 UTC m=+1516.944690782" watchObservedRunningTime="2026-07-28 17:34:28.671597303 +0000 UTC m=+1516.946213735"
Jul 28 17:34:29.650747 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:29.650716    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"
Jul 28 17:34:29.651894 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:29.651863    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv" podUID="083e2a6e-ffc7-407b-9b75-b26340519901" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 17:34:30.655195 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:30.655150    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv" podUID="083e2a6e-ffc7-407b-9b75-b26340519901" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 17:34:35.659481 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:35.659406    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"
Jul 28 17:34:35.659931 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:35.659874    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv" podUID="083e2a6e-ffc7-407b-9b75-b26340519901" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 17:34:45.660129 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:34:45.660096    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"
Jul 28 17:35:00.660075 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:00.660041    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"]
Jul 28 17:35:00.662786 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:00.660321    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv" podUID="083e2a6e-ffc7-407b-9b75-b26340519901" containerName="kserve-container" containerID="cri-o://8881c94fdee0fdd296ef2b8636114857a933277222ddb5dcf6ce48ffcf5071e6" gracePeriod=30
Jul 28 17:35:00.662786 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:00.660367    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv" podUID="083e2a6e-ffc7-407b-9b75-b26340519901" containerName="kube-rbac-proxy" containerID="cri-o://bba921d6597559828a38c6fbb92680e437f5ad23221d3d65185a09d7cc6b88c0" gracePeriod=30
Jul 28 17:35:00.741108 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:00.741074    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:01.773259 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:01.773224    2581 generic.go:356] "Generic (PLEG): container finished" podID="083e2a6e-ffc7-407b-9b75-b26340519901" containerID="bba921d6597559828a38c6fbb92680e437f5ad23221d3d65185a09d7cc6b88c0" exitCode=2
Jul 28 17:35:01.773628 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:01.773298    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv" event={"ID":"083e2a6e-ffc7-407b-9b75-b26340519901","Type":"ContainerDied","Data":"bba921d6597559828a38c6fbb92680e437f5ad23221d3d65185a09d7cc6b88c0"}
Jul 28 17:35:02.188848 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.188823    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"
Jul 28 17:35:02.286697 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.286663    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/083e2a6e-ffc7-407b-9b75-b26340519901-kserve-provision-location\") pod \"083e2a6e-ffc7-407b-9b75-b26340519901\" (UID: \"083e2a6e-ffc7-407b-9b75-b26340519901\") "
Jul 28 17:35:02.286697 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.286698    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/083e2a6e-ffc7-407b-9b75-b26340519901-cabundle-cert\") pod \"083e2a6e-ffc7-407b-9b75-b26340519901\" (UID: \"083e2a6e-ffc7-407b-9b75-b26340519901\") "
Jul 28 17:35:02.286925 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.286749    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x67hs\" (UniqueName: \"kubernetes.io/projected/083e2a6e-ffc7-407b-9b75-b26340519901-kube-api-access-x67hs\") pod \"083e2a6e-ffc7-407b-9b75-b26340519901\" (UID: \"083e2a6e-ffc7-407b-9b75-b26340519901\") "
Jul 28 17:35:02.286925 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.286770    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/083e2a6e-ffc7-407b-9b75-b26340519901-kube-rbac-proxy-sar-config\") pod \"083e2a6e-ffc7-407b-9b75-b26340519901\" (UID: \"083e2a6e-ffc7-407b-9b75-b26340519901\") "
Jul 28 17:35:02.286925 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.286793    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/083e2a6e-ffc7-407b-9b75-b26340519901-proxy-tls\") pod \"083e2a6e-ffc7-407b-9b75-b26340519901\" (UID: \"083e2a6e-ffc7-407b-9b75-b26340519901\") "
Jul 28 17:35:02.287182 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.287144    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/083e2a6e-ffc7-407b-9b75-b26340519901-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "083e2a6e-ffc7-407b-9b75-b26340519901" (UID: "083e2a6e-ffc7-407b-9b75-b26340519901"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:35:02.287290 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.287212    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/083e2a6e-ffc7-407b-9b75-b26340519901-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "083e2a6e-ffc7-407b-9b75-b26340519901" (UID: "083e2a6e-ffc7-407b-9b75-b26340519901"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:35:02.288012 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.287988    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083e2a6e-ffc7-407b-9b75-b26340519901-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "083e2a6e-ffc7-407b-9b75-b26340519901" (UID: "083e2a6e-ffc7-407b-9b75-b26340519901"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:35:02.288985 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.288961    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/083e2a6e-ffc7-407b-9b75-b26340519901-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "083e2a6e-ffc7-407b-9b75-b26340519901" (UID: "083e2a6e-ffc7-407b-9b75-b26340519901"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:35:02.289127 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.289108    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/083e2a6e-ffc7-407b-9b75-b26340519901-kube-api-access-x67hs" (OuterVolumeSpecName: "kube-api-access-x67hs") pod "083e2a6e-ffc7-407b-9b75-b26340519901" (UID: "083e2a6e-ffc7-407b-9b75-b26340519901"). InnerVolumeSpecName "kube-api-access-x67hs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:35:02.387662 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.387634    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/083e2a6e-ffc7-407b-9b75-b26340519901-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:35:02.387662 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.387660    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/083e2a6e-ffc7-407b-9b75-b26340519901-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:35:02.387818 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.387670    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/083e2a6e-ffc7-407b-9b75-b26340519901-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:35:02.387818 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.387679    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/083e2a6e-ffc7-407b-9b75-b26340519901-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:35:02.387818 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.387689    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x67hs\" (UniqueName: \"kubernetes.io/projected/083e2a6e-ffc7-407b-9b75-b26340519901-kube-api-access-x67hs\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:35:02.778376 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.778338    2581 generic.go:356] "Generic (PLEG): container finished" podID="083e2a6e-ffc7-407b-9b75-b26340519901" containerID="8881c94fdee0fdd296ef2b8636114857a933277222ddb5dcf6ce48ffcf5071e6" exitCode=0
Jul 28 17:35:02.778839 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.778402    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"
Jul 28 17:35:02.778839 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.778423    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv" event={"ID":"083e2a6e-ffc7-407b-9b75-b26340519901","Type":"ContainerDied","Data":"8881c94fdee0fdd296ef2b8636114857a933277222ddb5dcf6ce48ffcf5071e6"}
Jul 28 17:35:02.778839 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.778480    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv" event={"ID":"083e2a6e-ffc7-407b-9b75-b26340519901","Type":"ContainerDied","Data":"c6711565f01e4c1415f553b498260090fbeb4f48a5ee069e922585cac920de95"}
Jul 28 17:35:02.778839 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.778505    2581 scope.go:117] "RemoveContainer" containerID="bba921d6597559828a38c6fbb92680e437f5ad23221d3d65185a09d7cc6b88c0"
Jul 28 17:35:02.786038 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.786019    2581 scope.go:117] "RemoveContainer" containerID="8881c94fdee0fdd296ef2b8636114857a933277222ddb5dcf6ce48ffcf5071e6"
Jul 28 17:35:02.792753 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.792737    2581 scope.go:117] "RemoveContainer" containerID="038b2269749576dd1cb78609927c4c27bf8faa2304b9b837278f6947938becb8"
Jul 28 17:35:02.799371 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.799355    2581 scope.go:117] "RemoveContainer" containerID="bba921d6597559828a38c6fbb92680e437f5ad23221d3d65185a09d7cc6b88c0"
Jul 28 17:35:02.799618 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:35:02.799587    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bba921d6597559828a38c6fbb92680e437f5ad23221d3d65185a09d7cc6b88c0\": container with ID starting with bba921d6597559828a38c6fbb92680e437f5ad23221d3d65185a09d7cc6b88c0 not found: ID does not exist" containerID="bba921d6597559828a38c6fbb92680e437f5ad23221d3d65185a09d7cc6b88c0"
Jul 28 17:35:02.799665 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.799625    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bba921d6597559828a38c6fbb92680e437f5ad23221d3d65185a09d7cc6b88c0"} err="failed to get container status \"bba921d6597559828a38c6fbb92680e437f5ad23221d3d65185a09d7cc6b88c0\": rpc error: code = NotFound desc = could not find container \"bba921d6597559828a38c6fbb92680e437f5ad23221d3d65185a09d7cc6b88c0\": container with ID starting with bba921d6597559828a38c6fbb92680e437f5ad23221d3d65185a09d7cc6b88c0 not found: ID does not exist"
Jul 28 17:35:02.799665 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.799641    2581 scope.go:117] "RemoveContainer" containerID="8881c94fdee0fdd296ef2b8636114857a933277222ddb5dcf6ce48ffcf5071e6"
Jul 28 17:35:02.799843 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:35:02.799825    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8881c94fdee0fdd296ef2b8636114857a933277222ddb5dcf6ce48ffcf5071e6\": container with ID starting with 8881c94fdee0fdd296ef2b8636114857a933277222ddb5dcf6ce48ffcf5071e6 not found: ID does not exist" containerID="8881c94fdee0fdd296ef2b8636114857a933277222ddb5dcf6ce48ffcf5071e6"
Jul 28 17:35:02.799884 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.799848    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8881c94fdee0fdd296ef2b8636114857a933277222ddb5dcf6ce48ffcf5071e6"} err="failed to get container status \"8881c94fdee0fdd296ef2b8636114857a933277222ddb5dcf6ce48ffcf5071e6\": rpc error: code = NotFound desc = could not find container \"8881c94fdee0fdd296ef2b8636114857a933277222ddb5dcf6ce48ffcf5071e6\": container with ID starting with 8881c94fdee0fdd296ef2b8636114857a933277222ddb5dcf6ce48ffcf5071e6 not found: ID does not exist"
Jul 28 17:35:02.799884 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.799862    2581 scope.go:117] "RemoveContainer" containerID="038b2269749576dd1cb78609927c4c27bf8faa2304b9b837278f6947938becb8"
Jul 28 17:35:02.800111 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:35:02.800093    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"038b2269749576dd1cb78609927c4c27bf8faa2304b9b837278f6947938becb8\": container with ID starting with 038b2269749576dd1cb78609927c4c27bf8faa2304b9b837278f6947938becb8 not found: ID does not exist" containerID="038b2269749576dd1cb78609927c4c27bf8faa2304b9b837278f6947938becb8"
Jul 28 17:35:02.800177 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.800120    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"038b2269749576dd1cb78609927c4c27bf8faa2304b9b837278f6947938becb8"} err="failed to get container status \"038b2269749576dd1cb78609927c4c27bf8faa2304b9b837278f6947938becb8\": rpc error: code = NotFound desc = could not find container \"038b2269749576dd1cb78609927c4c27bf8faa2304b9b837278f6947938becb8\": container with ID starting with 038b2269749576dd1cb78609927c4c27bf8faa2304b9b837278f6947938becb8 not found: ID does not exist"
Jul 28 17:35:02.805037 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.805015    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"]
Jul 28 17:35:02.811085 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:02.811066    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-2njxv"]
Jul 28 17:35:04.289316 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:04.289285    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="083e2a6e-ffc7-407b-9b75-b26340519901" path="/var/lib/kubelet/pods/083e2a6e-ffc7-407b-9b75-b26340519901/volumes"
Jul 28 17:35:05.842423 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:05.842384    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:05.881072 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:05.881043    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"]
Jul 28 17:35:05.881335 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:05.881305    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="083e2a6e-ffc7-407b-9b75-b26340519901" containerName="storage-initializer"
Jul 28 17:35:05.881382 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:05.881336    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="083e2a6e-ffc7-407b-9b75-b26340519901" containerName="storage-initializer"
Jul 28 17:35:05.881382 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:05.881362    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="083e2a6e-ffc7-407b-9b75-b26340519901" containerName="kube-rbac-proxy"
Jul 28 17:35:05.881382 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:05.881368    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="083e2a6e-ffc7-407b-9b75-b26340519901" containerName="kube-rbac-proxy"
Jul 28 17:35:05.881382 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:05.881376    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="083e2a6e-ffc7-407b-9b75-b26340519901" containerName="kserve-container"
Jul 28 17:35:05.881382 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:05.881382    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="083e2a6e-ffc7-407b-9b75-b26340519901" containerName="kserve-container"
Jul 28 17:35:05.881544 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:05.881421    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="083e2a6e-ffc7-407b-9b75-b26340519901" containerName="kube-rbac-proxy"
Jul 28 17:35:05.881544 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:05.881430    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="083e2a6e-ffc7-407b-9b75-b26340519901" containerName="kserve-container"
Jul 28 17:35:05.886118 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:05.886100    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"
Jul 28 17:35:05.889185 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:05.889161    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:35:05.889337 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:05.889162    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Jul 28 17:35:05.889337 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:05.889251    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 17:35:05.889337 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:05.889326    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:35:05.889337 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:05.889338    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:35:05.889612 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:05.889371    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Jul 28 17:35:05.889612 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:05.889453    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:35:05.897639 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:05.897621    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"]
Jul 28 17:35:05.910906 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:05.910884    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58bda0e1-0a44-42b7-a657-62d0a8b0766d-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8\" (UID: \"58bda0e1-0a44-42b7-a657-62d0a8b0766d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"
Jul 28 17:35:05.911059 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:05.910927    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58bda0e1-0a44-42b7-a657-62d0a8b0766d-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8\" (UID: \"58bda0e1-0a44-42b7-a657-62d0a8b0766d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"
Jul 28 17:35:05.911059 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:05.910997    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xxl6\" (UniqueName: \"kubernetes.io/projected/58bda0e1-0a44-42b7-a657-62d0a8b0766d-kube-api-access-9xxl6\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8\" (UID: \"58bda0e1-0a44-42b7-a657-62d0a8b0766d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"
Jul 28 17:35:05.911059 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:05.911040    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58bda0e1-0a44-42b7-a657-62d0a8b0766d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8\" (UID: \"58bda0e1-0a44-42b7-a657-62d0a8b0766d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"
Jul 28 17:35:05.911202 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:05.911097    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58bda0e1-0a44-42b7-a657-62d0a8b0766d-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8\" (UID: \"58bda0e1-0a44-42b7-a657-62d0a8b0766d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"
Jul 28 17:35:06.012205 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:06.012178    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58bda0e1-0a44-42b7-a657-62d0a8b0766d-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8\" (UID: \"58bda0e1-0a44-42b7-a657-62d0a8b0766d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"
Jul 28 17:35:06.012341 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:06.012233    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58bda0e1-0a44-42b7-a657-62d0a8b0766d-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8\" (UID: \"58bda0e1-0a44-42b7-a657-62d0a8b0766d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"
Jul 28 17:35:06.012341 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:06.012275    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58bda0e1-0a44-42b7-a657-62d0a8b0766d-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8\" (UID: \"58bda0e1-0a44-42b7-a657-62d0a8b0766d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"
Jul 28 17:35:06.012341 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:06.012304    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xxl6\" (UniqueName: \"kubernetes.io/projected/58bda0e1-0a44-42b7-a657-62d0a8b0766d-kube-api-access-9xxl6\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8\" (UID: \"58bda0e1-0a44-42b7-a657-62d0a8b0766d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"
Jul 28 17:35:06.012519 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:06.012367    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58bda0e1-0a44-42b7-a657-62d0a8b0766d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8\" (UID: \"58bda0e1-0a44-42b7-a657-62d0a8b0766d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"
Jul 28 17:35:06.012519 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:35:06.012492    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-serving-cert: secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Jul 28 17:35:06.012646 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:35:06.012556    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58bda0e1-0a44-42b7-a657-62d0a8b0766d-proxy-tls podName:58bda0e1-0a44-42b7-a657-62d0a8b0766d nodeName:}" failed. No retries permitted until 2026-07-28 17:35:06.512536377 +0000 UTC m=+1554.787152788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/58bda0e1-0a44-42b7-a657-62d0a8b0766d-proxy-tls") pod "isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8" (UID: "58bda0e1-0a44-42b7-a657-62d0a8b0766d") : secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Jul 28 17:35:06.012646 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:06.012663    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58bda0e1-0a44-42b7-a657-62d0a8b0766d-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8\" (UID: \"58bda0e1-0a44-42b7-a657-62d0a8b0766d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"
Jul 28 17:35:06.012901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:06.012883    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58bda0e1-0a44-42b7-a657-62d0a8b0766d-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8\" (UID: \"58bda0e1-0a44-42b7-a657-62d0a8b0766d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"
Jul 28 17:35:06.013011 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:06.012995    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58bda0e1-0a44-42b7-a657-62d0a8b0766d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8\" (UID: \"58bda0e1-0a44-42b7-a657-62d0a8b0766d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"
Jul 28 17:35:06.022896 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:06.022874    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xxl6\" (UniqueName: \"kubernetes.io/projected/58bda0e1-0a44-42b7-a657-62d0a8b0766d-kube-api-access-9xxl6\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8\" (UID: \"58bda0e1-0a44-42b7-a657-62d0a8b0766d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"
Jul 28 17:35:06.516060 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:06.516022    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58bda0e1-0a44-42b7-a657-62d0a8b0766d-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8\" (UID: \"58bda0e1-0a44-42b7-a657-62d0a8b0766d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"
Jul 28 17:35:06.518332 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:06.518313    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58bda0e1-0a44-42b7-a657-62d0a8b0766d-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8\" (UID: \"58bda0e1-0a44-42b7-a657-62d0a8b0766d\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"
Jul 28 17:35:06.796660 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:06.796585    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"
Jul 28 17:35:06.921203 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:06.921172    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"]
Jul 28 17:35:06.924165 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:35:06.924138    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58bda0e1_0a44_42b7_a657_62d0a8b0766d.slice/crio-f9d078c51bdd59a552cf2cf6c25405359f7d23d9313f8e5503a5c5442bbf3a3e WatchSource:0}: Error finding container f9d078c51bdd59a552cf2cf6c25405359f7d23d9313f8e5503a5c5442bbf3a3e: Status 404 returned error can't find the container with id f9d078c51bdd59a552cf2cf6c25405359f7d23d9313f8e5503a5c5442bbf3a3e
Jul 28 17:35:07.798846 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:07.798814    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8" event={"ID":"58bda0e1-0a44-42b7-a657-62d0a8b0766d","Type":"ContainerStarted","Data":"709977f0d05e570ce84a806c6ec37df4464419ef198746dfe55867b80a53b859"}
Jul 28 17:35:07.799041 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:07.798872    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8" event={"ID":"58bda0e1-0a44-42b7-a657-62d0a8b0766d","Type":"ContainerStarted","Data":"f9d078c51bdd59a552cf2cf6c25405359f7d23d9313f8e5503a5c5442bbf3a3e"}
Jul 28 17:35:07.922854 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:07.922823    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:11.814686 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:11.814651    2581 generic.go:356] "Generic (PLEG): container finished" podID="58bda0e1-0a44-42b7-a657-62d0a8b0766d" containerID="709977f0d05e570ce84a806c6ec37df4464419ef198746dfe55867b80a53b859" exitCode=0
Jul 28 17:35:11.815199 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:11.814730    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8" event={"ID":"58bda0e1-0a44-42b7-a657-62d0a8b0766d","Type":"ContainerDied","Data":"709977f0d05e570ce84a806c6ec37df4464419ef198746dfe55867b80a53b859"}
Jul 28 17:35:12.819904 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:12.819870    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8" event={"ID":"58bda0e1-0a44-42b7-a657-62d0a8b0766d","Type":"ContainerStarted","Data":"caf4a21246509957f9195a417833f679f01526197b1b3a11663d263db026f480"}
Jul 28 17:35:12.820314 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:12.819929    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8" event={"ID":"58bda0e1-0a44-42b7-a657-62d0a8b0766d","Type":"ContainerStarted","Data":"1b30bb0ad4a10fc99bd411eddede4627016c5d26445f40b915068db432c50361"}
Jul 28 17:35:12.820314 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:12.820144    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"
Jul 28 17:35:12.851363 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:12.851303    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8" podStartSLOduration=3.472913422 podStartE2EDuration="7.851284952s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.37837153s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:12.850875533 +0000 UTC m=+1561.125491966" watchObservedRunningTime="2026-07-28 17:35:12.851284952 +0000 UTC m=+1561.125901386"
Jul 28 17:35:13.824123 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:13.824085    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"
Jul 28 17:35:13.825573 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:13.825539    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8" podUID="58bda0e1-0a44-42b7-a657-62d0a8b0766d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 17:35:14.827777 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:14.827737    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8" podUID="58bda0e1-0a44-42b7-a657-62d0a8b0766d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 17:35:19.832304 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:19.832268    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"
Jul 28 17:35:19.832869 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:19.832842    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8" podUID="58bda0e1-0a44-42b7-a657-62d0a8b0766d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 17:35:29.833646 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:29.833616    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"
Jul 28 17:35:40.947607 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:40.947576    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"]
Jul 28 17:35:40.948114 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:40.947818    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8" podUID="58bda0e1-0a44-42b7-a657-62d0a8b0766d" containerName="kserve-container" containerID="cri-o://1b30bb0ad4a10fc99bd411eddede4627016c5d26445f40b915068db432c50361" gracePeriod=30
Jul 28 17:35:40.948114 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:40.947867    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8" podUID="58bda0e1-0a44-42b7-a657-62d0a8b0766d" containerName="kube-rbac-proxy" containerID="cri-o://caf4a21246509957f9195a417833f679f01526197b1b3a11663d263db026f480" gracePeriod=30
Jul 28 17:35:40.970000 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:40.969927    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:41.925674 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:41.925637    2581 generic.go:356] "Generic (PLEG): container finished" podID="58bda0e1-0a44-42b7-a657-62d0a8b0766d" containerID="caf4a21246509957f9195a417833f679f01526197b1b3a11663d263db026f480" exitCode=2
Jul 28 17:35:41.925878 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:41.925711    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8" event={"ID":"58bda0e1-0a44-42b7-a657-62d0a8b0766d","Type":"ContainerDied","Data":"caf4a21246509957f9195a417833f679f01526197b1b3a11663d263db026f480"}
Jul 28 17:35:42.187207 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.187155    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"
Jul 28 17:35:42.261917 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.261889    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58bda0e1-0a44-42b7-a657-62d0a8b0766d-kube-rbac-proxy-sar-config\") pod \"58bda0e1-0a44-42b7-a657-62d0a8b0766d\" (UID: \"58bda0e1-0a44-42b7-a657-62d0a8b0766d\") "
Jul 28 17:35:42.262071 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.261927    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58bda0e1-0a44-42b7-a657-62d0a8b0766d-proxy-tls\") pod \"58bda0e1-0a44-42b7-a657-62d0a8b0766d\" (UID: \"58bda0e1-0a44-42b7-a657-62d0a8b0766d\") "
Jul 28 17:35:42.262071 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.261968    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58bda0e1-0a44-42b7-a657-62d0a8b0766d-cabundle-cert\") pod \"58bda0e1-0a44-42b7-a657-62d0a8b0766d\" (UID: \"58bda0e1-0a44-42b7-a657-62d0a8b0766d\") "
Jul 28 17:35:42.262071 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.262007    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xxl6\" (UniqueName: \"kubernetes.io/projected/58bda0e1-0a44-42b7-a657-62d0a8b0766d-kube-api-access-9xxl6\") pod \"58bda0e1-0a44-42b7-a657-62d0a8b0766d\" (UID: \"58bda0e1-0a44-42b7-a657-62d0a8b0766d\") "
Jul 28 17:35:42.262071 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.262045    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58bda0e1-0a44-42b7-a657-62d0a8b0766d-kserve-provision-location\") pod \"58bda0e1-0a44-42b7-a657-62d0a8b0766d\" (UID: \"58bda0e1-0a44-42b7-a657-62d0a8b0766d\") "
Jul 28 17:35:42.262339 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.262279    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58bda0e1-0a44-42b7-a657-62d0a8b0766d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "58bda0e1-0a44-42b7-a657-62d0a8b0766d" (UID: "58bda0e1-0a44-42b7-a657-62d0a8b0766d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:35:42.262403 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.262374    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58bda0e1-0a44-42b7-a657-62d0a8b0766d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "58bda0e1-0a44-42b7-a657-62d0a8b0766d" (UID: "58bda0e1-0a44-42b7-a657-62d0a8b0766d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:35:42.263090 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.263070    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58bda0e1-0a44-42b7-a657-62d0a8b0766d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "58bda0e1-0a44-42b7-a657-62d0a8b0766d" (UID: "58bda0e1-0a44-42b7-a657-62d0a8b0766d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:35:42.263843 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.263822    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58bda0e1-0a44-42b7-a657-62d0a8b0766d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "58bda0e1-0a44-42b7-a657-62d0a8b0766d" (UID: "58bda0e1-0a44-42b7-a657-62d0a8b0766d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:35:42.264135 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.264111    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58bda0e1-0a44-42b7-a657-62d0a8b0766d-kube-api-access-9xxl6" (OuterVolumeSpecName: "kube-api-access-9xxl6") pod "58bda0e1-0a44-42b7-a657-62d0a8b0766d" (UID: "58bda0e1-0a44-42b7-a657-62d0a8b0766d"). InnerVolumeSpecName "kube-api-access-9xxl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:35:42.363471 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.363445    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xxl6\" (UniqueName: \"kubernetes.io/projected/58bda0e1-0a44-42b7-a657-62d0a8b0766d-kube-api-access-9xxl6\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:35:42.363471 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.363467    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58bda0e1-0a44-42b7-a657-62d0a8b0766d-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:35:42.363610 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.363476    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58bda0e1-0a44-42b7-a657-62d0a8b0766d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:35:42.363610 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.363485    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58bda0e1-0a44-42b7-a657-62d0a8b0766d-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:35:42.363610 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.363494    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58bda0e1-0a44-42b7-a657-62d0a8b0766d-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:35:42.930248 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.930211    2581 generic.go:356] "Generic (PLEG): container finished" podID="58bda0e1-0a44-42b7-a657-62d0a8b0766d" containerID="1b30bb0ad4a10fc99bd411eddede4627016c5d26445f40b915068db432c50361" exitCode=0
Jul 28 17:35:42.930424 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.930271    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"
Jul 28 17:35:42.930424 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.930289    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8" event={"ID":"58bda0e1-0a44-42b7-a657-62d0a8b0766d","Type":"ContainerDied","Data":"1b30bb0ad4a10fc99bd411eddede4627016c5d26445f40b915068db432c50361"}
Jul 28 17:35:42.930424 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.930361    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8" event={"ID":"58bda0e1-0a44-42b7-a657-62d0a8b0766d","Type":"ContainerDied","Data":"f9d078c51bdd59a552cf2cf6c25405359f7d23d9313f8e5503a5c5442bbf3a3e"}
Jul 28 17:35:42.930424 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.930402    2581 scope.go:117] "RemoveContainer" containerID="caf4a21246509957f9195a417833f679f01526197b1b3a11663d263db026f480"
Jul 28 17:35:42.937996 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.937979    2581 scope.go:117] "RemoveContainer" containerID="1b30bb0ad4a10fc99bd411eddede4627016c5d26445f40b915068db432c50361"
Jul 28 17:35:42.944467 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.944450    2581 scope.go:117] "RemoveContainer" containerID="709977f0d05e570ce84a806c6ec37df4464419ef198746dfe55867b80a53b859"
Jul 28 17:35:42.948994 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.948975    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"]
Jul 28 17:35:42.951286 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.951272    2581 scope.go:117] "RemoveContainer" containerID="caf4a21246509957f9195a417833f679f01526197b1b3a11663d263db026f480"
Jul 28 17:35:42.951625 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:35:42.951603    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caf4a21246509957f9195a417833f679f01526197b1b3a11663d263db026f480\": container with ID starting with caf4a21246509957f9195a417833f679f01526197b1b3a11663d263db026f480 not found: ID does not exist" containerID="caf4a21246509957f9195a417833f679f01526197b1b3a11663d263db026f480"
Jul 28 17:35:42.951763 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.951636    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caf4a21246509957f9195a417833f679f01526197b1b3a11663d263db026f480"} err="failed to get container status \"caf4a21246509957f9195a417833f679f01526197b1b3a11663d263db026f480\": rpc error: code = NotFound desc = could not find container \"caf4a21246509957f9195a417833f679f01526197b1b3a11663d263db026f480\": container with ID starting with caf4a21246509957f9195a417833f679f01526197b1b3a11663d263db026f480 not found: ID does not exist"
Jul 28 17:35:42.951763 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.951659    2581 scope.go:117] "RemoveContainer" containerID="1b30bb0ad4a10fc99bd411eddede4627016c5d26445f40b915068db432c50361"
Jul 28 17:35:42.952052 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:35:42.951984    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b30bb0ad4a10fc99bd411eddede4627016c5d26445f40b915068db432c50361\": container with ID starting with 1b30bb0ad4a10fc99bd411eddede4627016c5d26445f40b915068db432c50361 not found: ID does not exist" containerID="1b30bb0ad4a10fc99bd411eddede4627016c5d26445f40b915068db432c50361"
Jul 28 17:35:42.952180 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.952017    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b30bb0ad4a10fc99bd411eddede4627016c5d26445f40b915068db432c50361"} err="failed to get container status \"1b30bb0ad4a10fc99bd411eddede4627016c5d26445f40b915068db432c50361\": rpc error: code = NotFound desc = could not find container \"1b30bb0ad4a10fc99bd411eddede4627016c5d26445f40b915068db432c50361\": container with ID starting with 1b30bb0ad4a10fc99bd411eddede4627016c5d26445f40b915068db432c50361 not found: ID does not exist"
Jul 28 17:35:42.952180 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.952149    2581 scope.go:117] "RemoveContainer" containerID="709977f0d05e570ce84a806c6ec37df4464419ef198746dfe55867b80a53b859"
Jul 28 17:35:42.952569 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:35:42.952533    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"709977f0d05e570ce84a806c6ec37df4464419ef198746dfe55867b80a53b859\": container with ID starting with 709977f0d05e570ce84a806c6ec37df4464419ef198746dfe55867b80a53b859 not found: ID does not exist" containerID="709977f0d05e570ce84a806c6ec37df4464419ef198746dfe55867b80a53b859"
Jul 28 17:35:42.952650 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.952567    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"709977f0d05e570ce84a806c6ec37df4464419ef198746dfe55867b80a53b859"} err="failed to get container status \"709977f0d05e570ce84a806c6ec37df4464419ef198746dfe55867b80a53b859\": rpc error: code = NotFound desc = could not find container \"709977f0d05e570ce84a806c6ec37df4464419ef198746dfe55867b80a53b859\": container with ID starting with 709977f0d05e570ce84a806c6ec37df4464419ef198746dfe55867b80a53b859 not found: ID does not exist"
Jul 28 17:35:42.953205 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:42.953182    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-qwvn8"]
Jul 28 17:35:44.289531 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:44.289499    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58bda0e1-0a44-42b7-a657-62d0a8b0766d" path="/var/lib/kubelet/pods/58bda0e1-0a44-42b7-a657-62d0a8b0766d/volumes"
Jul 28 17:35:46.099151 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.099124    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:46.138458 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.138427    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"]
Jul 28 17:35:46.138676 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.138664    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58bda0e1-0a44-42b7-a657-62d0a8b0766d" containerName="storage-initializer"
Jul 28 17:35:46.138721 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.138677    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="58bda0e1-0a44-42b7-a657-62d0a8b0766d" containerName="storage-initializer"
Jul 28 17:35:46.138721 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.138685    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58bda0e1-0a44-42b7-a657-62d0a8b0766d" containerName="kserve-container"
Jul 28 17:35:46.138721 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.138691    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="58bda0e1-0a44-42b7-a657-62d0a8b0766d" containerName="kserve-container"
Jul 28 17:35:46.138721 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.138699    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58bda0e1-0a44-42b7-a657-62d0a8b0766d" containerName="kube-rbac-proxy"
Jul 28 17:35:46.138721 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.138705    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="58bda0e1-0a44-42b7-a657-62d0a8b0766d" containerName="kube-rbac-proxy"
Jul 28 17:35:46.138874 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.138746    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="58bda0e1-0a44-42b7-a657-62d0a8b0766d" containerName="kserve-container"
Jul 28 17:35:46.138874 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.138755    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="58bda0e1-0a44-42b7-a657-62d0a8b0766d" containerName="kube-rbac-proxy"
Jul 28 17:35:46.141682 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.141665    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"
Jul 28 17:35:46.144403 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.144378    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:35:46.144543 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.144523    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Jul 28 17:35:46.144608 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.144549    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:35:46.144608 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.144588    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:35:46.144713 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.144604    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 17:35:46.144713 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.144447    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Jul 28 17:35:46.144713 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.144534    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:35:46.152401 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.152381    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"]
Jul 28 17:35:46.158876 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.158852    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:46.175188 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.175163    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"]
Jul 28 17:35:46.182419 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.182394    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"
Jul 28 17:35:46.185106 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.185087    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Jul 28 17:35:46.187164 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.187120    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c315a841-b6b4-4b79-ab60-2f4658c46a87-kserve-provision-location\") pod \"isvc-canary-create-predictor-594f5f9bd5-rlg8t\" (UID: \"c315a841-b6b4-4b79-ab60-2f4658c46a87\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"
Jul 28 17:35:46.187326 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.187204    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c315a841-b6b4-4b79-ab60-2f4658c46a87-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-594f5f9bd5-rlg8t\" (UID: \"c315a841-b6b4-4b79-ab60-2f4658c46a87\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"
Jul 28 17:35:46.187326 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.187240    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lqp4\" (UniqueName: \"kubernetes.io/projected/c315a841-b6b4-4b79-ab60-2f4658c46a87-kube-api-access-6lqp4\") pod \"isvc-canary-create-predictor-594f5f9bd5-rlg8t\" (UID: \"c315a841-b6b4-4b79-ab60-2f4658c46a87\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"
Jul 28 17:35:46.187326 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.187271    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c315a841-b6b4-4b79-ab60-2f4658c46a87-proxy-tls\") pod \"isvc-canary-create-predictor-594f5f9bd5-rlg8t\" (UID: \"c315a841-b6b4-4b79-ab60-2f4658c46a87\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"
Jul 28 17:35:46.188324 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.187355    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c315a841-b6b4-4b79-ab60-2f4658c46a87-cabundle-cert\") pod \"isvc-canary-create-predictor-594f5f9bd5-rlg8t\" (UID: \"c315a841-b6b4-4b79-ab60-2f4658c46a87\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"
Jul 28 17:35:46.189839 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.189818    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"]
Jul 28 17:35:46.287962 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.287914    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c315a841-b6b4-4b79-ab60-2f4658c46a87-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-594f5f9bd5-rlg8t\" (UID: \"c315a841-b6b4-4b79-ab60-2f4658c46a87\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"
Jul 28 17:35:46.288158 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.287974    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5b976d889b-5rq4l\" (UID: \"ef517b9c-a92a-4148-ab96-64e7dfa3bca4\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"
Jul 28 17:35:46.288158 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.288007    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lqp4\" (UniqueName: \"kubernetes.io/projected/c315a841-b6b4-4b79-ab60-2f4658c46a87-kube-api-access-6lqp4\") pod \"isvc-canary-create-predictor-594f5f9bd5-rlg8t\" (UID: \"c315a841-b6b4-4b79-ab60-2f4658c46a87\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"
Jul 28 17:35:46.288158 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.288025    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5b976d889b-5rq4l\" (UID: \"ef517b9c-a92a-4148-ab96-64e7dfa3bca4\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"
Jul 28 17:35:46.288158 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.288051    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c315a841-b6b4-4b79-ab60-2f4658c46a87-proxy-tls\") pod \"isvc-canary-create-predictor-594f5f9bd5-rlg8t\" (UID: \"c315a841-b6b4-4b79-ab60-2f4658c46a87\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"
Jul 28 17:35:46.288158 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.288098    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c315a841-b6b4-4b79-ab60-2f4658c46a87-cabundle-cert\") pod \"isvc-canary-create-predictor-594f5f9bd5-rlg8t\" (UID: \"c315a841-b6b4-4b79-ab60-2f4658c46a87\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"
Jul 28 17:35:46.288158 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.288116    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5b976d889b-5rq4l\" (UID: \"ef517b9c-a92a-4148-ab96-64e7dfa3bca4\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"
Jul 28 17:35:46.288158 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.288139    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c315a841-b6b4-4b79-ab60-2f4658c46a87-kserve-provision-location\") pod \"isvc-canary-create-predictor-594f5f9bd5-rlg8t\" (UID: \"c315a841-b6b4-4b79-ab60-2f4658c46a87\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"
Jul 28 17:35:46.288535 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.288174    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rrz8\" (UniqueName: \"kubernetes.io/projected/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-kube-api-access-7rrz8\") pod \"isvc-canary-create-v2-predictor-5b976d889b-5rq4l\" (UID: \"ef517b9c-a92a-4148-ab96-64e7dfa3bca4\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"
Jul 28 17:35:46.288535 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.288192    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5b976d889b-5rq4l\" (UID: \"ef517b9c-a92a-4148-ab96-64e7dfa3bca4\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"
Jul 28 17:35:46.288535 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:35:46.288250    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Jul 28 17:35:46.288535 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:35:46.288319    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c315a841-b6b4-4b79-ab60-2f4658c46a87-proxy-tls podName:c315a841-b6b4-4b79-ab60-2f4658c46a87 nodeName:}" failed. No retries permitted until 2026-07-28 17:35:46.788300171 +0000 UTC m=+1595.062916586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c315a841-b6b4-4b79-ab60-2f4658c46a87-proxy-tls") pod "isvc-canary-create-predictor-594f5f9bd5-rlg8t" (UID: "c315a841-b6b4-4b79-ab60-2f4658c46a87") : secret "isvc-canary-create-predictor-serving-cert" not found
Jul 28 17:35:46.288750 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.288702    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c315a841-b6b4-4b79-ab60-2f4658c46a87-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-594f5f9bd5-rlg8t\" (UID: \"c315a841-b6b4-4b79-ab60-2f4658c46a87\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"
Jul 28 17:35:46.288806 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.288779    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c315a841-b6b4-4b79-ab60-2f4658c46a87-cabundle-cert\") pod \"isvc-canary-create-predictor-594f5f9bd5-rlg8t\" (UID: \"c315a841-b6b4-4b79-ab60-2f4658c46a87\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"
Jul 28 17:35:46.288861 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.288816    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c315a841-b6b4-4b79-ab60-2f4658c46a87-kserve-provision-location\") pod \"isvc-canary-create-predictor-594f5f9bd5-rlg8t\" (UID: \"c315a841-b6b4-4b79-ab60-2f4658c46a87\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"
Jul 28 17:35:46.296803 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.296772    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lqp4\" (UniqueName: \"kubernetes.io/projected/c315a841-b6b4-4b79-ab60-2f4658c46a87-kube-api-access-6lqp4\") pod \"isvc-canary-create-predictor-594f5f9bd5-rlg8t\" (UID: \"c315a841-b6b4-4b79-ab60-2f4658c46a87\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"
Jul 28 17:35:46.389545 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.389454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rrz8\" (UniqueName: \"kubernetes.io/projected/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-kube-api-access-7rrz8\") pod \"isvc-canary-create-v2-predictor-5b976d889b-5rq4l\" (UID: \"ef517b9c-a92a-4148-ab96-64e7dfa3bca4\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"
Jul 28 17:35:46.389545 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.389495    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5b976d889b-5rq4l\" (UID: \"ef517b9c-a92a-4148-ab96-64e7dfa3bca4\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"
Jul 28 17:35:46.389782 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.389614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5b976d889b-5rq4l\" (UID: \"ef517b9c-a92a-4148-ab96-64e7dfa3bca4\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"
Jul 28 17:35:46.389782 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.389661    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5b976d889b-5rq4l\" (UID: \"ef517b9c-a92a-4148-ab96-64e7dfa3bca4\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"
Jul 28 17:35:46.389782 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.389754    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5b976d889b-5rq4l\" (UID: \"ef517b9c-a92a-4148-ab96-64e7dfa3bca4\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"
Jul 28 17:35:46.390081 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.390058    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5b976d889b-5rq4l\" (UID: \"ef517b9c-a92a-4148-ab96-64e7dfa3bca4\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"
Jul 28 17:35:46.390411 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.390392    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5b976d889b-5rq4l\" (UID: \"ef517b9c-a92a-4148-ab96-64e7dfa3bca4\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"
Jul 28 17:35:46.390470 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.390444    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5b976d889b-5rq4l\" (UID: \"ef517b9c-a92a-4148-ab96-64e7dfa3bca4\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"
Jul 28 17:35:46.392122 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.392090    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5b976d889b-5rq4l\" (UID: \"ef517b9c-a92a-4148-ab96-64e7dfa3bca4\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"
Jul 28 17:35:46.399883 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.399865    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rrz8\" (UniqueName: \"kubernetes.io/projected/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-kube-api-access-7rrz8\") pod \"isvc-canary-create-v2-predictor-5b976d889b-5rq4l\" (UID: \"ef517b9c-a92a-4148-ab96-64e7dfa3bca4\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"
Jul 28 17:35:46.413405 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.413384    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:46.493723 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.493694    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"
Jul 28 17:35:46.628593 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.628558    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"]
Jul 28 17:35:46.631687 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:35:46.631665    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef517b9c_a92a_4148_ab96_64e7dfa3bca4.slice/crio-718855b9ba6ec4b2701d95fbec38d0d66556cb5e27d6fcd5584a311fe0e7171a WatchSource:0}: Error finding container 718855b9ba6ec4b2701d95fbec38d0d66556cb5e27d6fcd5584a311fe0e7171a: Status 404 returned error can't find the container with id 718855b9ba6ec4b2701d95fbec38d0d66556cb5e27d6fcd5584a311fe0e7171a
Jul 28 17:35:46.657492 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.657454    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:35:46.793222 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.793180    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c315a841-b6b4-4b79-ab60-2f4658c46a87-proxy-tls\") pod \"isvc-canary-create-predictor-594f5f9bd5-rlg8t\" (UID: \"c315a841-b6b4-4b79-ab60-2f4658c46a87\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"
Jul 28 17:35:46.795515 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.795491    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c315a841-b6b4-4b79-ab60-2f4658c46a87-proxy-tls\") pod \"isvc-canary-create-predictor-594f5f9bd5-rlg8t\" (UID: \"c315a841-b6b4-4b79-ab60-2f4658c46a87\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"
Jul 28 17:35:46.947043 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.947003    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l" event={"ID":"ef517b9c-a92a-4148-ab96-64e7dfa3bca4","Type":"ContainerStarted","Data":"f49dd50b37317eaea5ca438b5cc7234c99e5e6b5fd090bf9b7a79b6d4411c034"}
Jul 28 17:35:46.947243 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:46.947061    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l" event={"ID":"ef517b9c-a92a-4148-ab96-64e7dfa3bca4","Type":"ContainerStarted","Data":"718855b9ba6ec4b2701d95fbec38d0d66556cb5e27d6fcd5584a311fe0e7171a"}
Jul 28 17:35:47.053464 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:47.053421    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"
Jul 28 17:35:47.187909 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:47.187875    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"]
Jul 28 17:35:47.190696 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:35:47.190672    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc315a841_b6b4_4b79_ab60_2f4658c46a87.slice/crio-ac25fee2a34af792a04e9aff142fcb4e537ead869db05fcecc490c3d4ed2acc9 WatchSource:0}: Error finding container ac25fee2a34af792a04e9aff142fcb4e537ead869db05fcecc490c3d4ed2acc9: Status 404 returned error can't find the container with id ac25fee2a34af792a04e9aff142fcb4e537ead869db05fcecc490c3d4ed2acc9
Jul 28 17:35:47.951871 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:47.951830    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t" event={"ID":"c315a841-b6b4-4b79-ab60-2f4658c46a87","Type":"ContainerStarted","Data":"d3421c8db888f1a5ac9306d0333689d1b82d7524e9fe619321cfcf9435061126"}
Jul 28 17:35:47.952081 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:47.951908    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t" event={"ID":"c315a841-b6b4-4b79-ab60-2f4658c46a87","Type":"ContainerStarted","Data":"ac25fee2a34af792a04e9aff142fcb4e537ead869db05fcecc490c3d4ed2acc9"}
Jul 28 17:35:50.963844 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:50.963814    2581 generic.go:356] "Generic (PLEG): container finished" podID="ef517b9c-a92a-4148-ab96-64e7dfa3bca4" containerID="f49dd50b37317eaea5ca438b5cc7234c99e5e6b5fd090bf9b7a79b6d4411c034" exitCode=0
Jul 28 17:35:50.964192 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:50.963885    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l" event={"ID":"ef517b9c-a92a-4148-ab96-64e7dfa3bca4","Type":"ContainerDied","Data":"f49dd50b37317eaea5ca438b5cc7234c99e5e6b5fd090bf9b7a79b6d4411c034"}
Jul 28 17:35:50.965473 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:50.965451    2581 generic.go:356] "Generic (PLEG): container finished" podID="c315a841-b6b4-4b79-ab60-2f4658c46a87" containerID="d3421c8db888f1a5ac9306d0333689d1b82d7524e9fe619321cfcf9435061126" exitCode=0
Jul 28 17:35:50.965548 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:50.965480    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t" event={"ID":"c315a841-b6b4-4b79-ab60-2f4658c46a87","Type":"ContainerDied","Data":"d3421c8db888f1a5ac9306d0333689d1b82d7524e9fe619321cfcf9435061126"}
Jul 28 17:35:51.970891 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:51.970847    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l" event={"ID":"ef517b9c-a92a-4148-ab96-64e7dfa3bca4","Type":"ContainerStarted","Data":"0df9c5c54dec281fdece4e6f4dca89ef6f568b6cf75ebbbb54d7ccbc90680d6f"}
Jul 28 17:35:51.971322 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:51.970966    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l" event={"ID":"ef517b9c-a92a-4148-ab96-64e7dfa3bca4","Type":"ContainerStarted","Data":"2df264421e8de271c2176880030eb833fe0549cb1882c5dda0cfc9ee742bc9df"}
Jul 28 17:35:51.971375 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:51.971320    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"
Jul 28 17:35:51.971469 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:51.971448    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"
Jul 28 17:35:51.972869 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:51.972834    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t" event={"ID":"c315a841-b6b4-4b79-ab60-2f4658c46a87","Type":"ContainerStarted","Data":"5b5d66374c73638a5c970b2ed559377ad3244ad034e4492392e45743a53c5a24"}
Jul 28 17:35:51.973003 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:51.972905    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t" event={"ID":"c315a841-b6b4-4b79-ab60-2f4658c46a87","Type":"ContainerStarted","Data":"bdc31b4f3c3e86a8639ab68da9ca8e85e2a6c1f909a0cb72c1905b93a63b433a"}
Jul 28 17:35:51.973070 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:51.973010    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l" podUID="ef517b9c-a92a-4148-ab96-64e7dfa3bca4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:35:51.973269 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:51.973252    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"
Jul 28 17:35:51.973341 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:51.973299    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"
Jul 28 17:35:51.974274 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:51.974253    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t" podUID="c315a841-b6b4-4b79-ab60-2f4658c46a87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 17:35:51.992738 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:51.992687    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l" podStartSLOduration=2.343922891 podStartE2EDuration="5.992673712s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.648750821s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:51.991618321 +0000 UTC m=+1600.266234827" watchObservedRunningTime="2026-07-28 17:35:51.992673712 +0000 UTC m=+1600.267290144"
Jul 28 17:35:52.011407 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:52.011367    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t" podStartSLOduration=2.377553583 podStartE2EDuration="6.011351481s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.633797898s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:52.011078701 +0000 UTC m=+1600.285695134" watchObservedRunningTime="2026-07-28 17:35:52.011351481 +0000 UTC m=+1600.285967916"
Jul 28 17:35:52.976319 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:52.976270    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l" podUID="ef517b9c-a92a-4148-ab96-64e7dfa3bca4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:35:52.976759 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:52.976382    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t" podUID="c315a841-b6b4-4b79-ab60-2f4658c46a87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 17:35:53.979794 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:53.979753    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l" podUID="ef517b9c-a92a-4148-ab96-64e7dfa3bca4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:35:57.980765 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:57.980736    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"
Jul 28 17:35:57.981372 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:57.981346    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t" podUID="c315a841-b6b4-4b79-ab60-2f4658c46a87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 17:35:58.984673 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:58.984642    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"
Jul 28 17:35:58.985346 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:35:58.985315    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l" podUID="ef517b9c-a92a-4148-ab96-64e7dfa3bca4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:36:07.981547 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:36:07.981466    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t" podUID="c315a841-b6b4-4b79-ab60-2f4658c46a87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 17:36:08.986196 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:36:08.986154    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l" podUID="ef517b9c-a92a-4148-ab96-64e7dfa3bca4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:36:17.981892 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:36:17.981850    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t" podUID="c315a841-b6b4-4b79-ab60-2f4658c46a87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 17:36:18.985684 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:36:18.985642    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l" podUID="ef517b9c-a92a-4148-ab96-64e7dfa3bca4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:36:27.982273 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:36:27.982232    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t" podUID="c315a841-b6b4-4b79-ab60-2f4658c46a87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 17:36:28.986175 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:36:28.986137    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l" podUID="ef517b9c-a92a-4148-ab96-64e7dfa3bca4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:36:37.981388 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:36:37.981349    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t" podUID="c315a841-b6b4-4b79-ab60-2f4658c46a87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 17:36:38.985209 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:36:38.985171    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l" podUID="ef517b9c-a92a-4148-ab96-64e7dfa3bca4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:36:47.982037 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:36:47.981991    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t" podUID="c315a841-b6b4-4b79-ab60-2f4658c46a87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 17:36:48.986174 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:36:48.986134    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l" podUID="ef517b9c-a92a-4148-ab96-64e7dfa3bca4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:36:57.981687 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:36:57.981643    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t" podUID="c315a841-b6b4-4b79-ab60-2f4658c46a87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 17:36:58.985640 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:36:58.985599    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l" podUID="ef517b9c-a92a-4148-ab96-64e7dfa3bca4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:37:07.982144 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:07.982109    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"
Jul 28 17:37:08.985825 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:08.985791    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"
Jul 28 17:37:16.282446 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:16.282410    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"]
Jul 28 17:37:16.283039 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:16.282692    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l" podUID="ef517b9c-a92a-4148-ab96-64e7dfa3bca4" containerName="kserve-container" containerID="cri-o://2df264421e8de271c2176880030eb833fe0549cb1882c5dda0cfc9ee742bc9df" gracePeriod=30
Jul 28 17:37:16.283039 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:16.282702    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l" podUID="ef517b9c-a92a-4148-ab96-64e7dfa3bca4" containerName="kube-rbac-proxy" containerID="cri-o://0df9c5c54dec281fdece4e6f4dca89ef6f568b6cf75ebbbb54d7ccbc90680d6f" gracePeriod=30
Jul 28 17:37:16.291375 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:16.291353    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"]
Jul 28 17:37:16.291590 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:16.291569    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t" podUID="c315a841-b6b4-4b79-ab60-2f4658c46a87" containerName="kube-rbac-proxy" containerID="cri-o://5b5d66374c73638a5c970b2ed559377ad3244ad034e4492392e45743a53c5a24" gracePeriod=30
Jul 28 17:37:16.291660 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:16.291561    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t" podUID="c315a841-b6b4-4b79-ab60-2f4658c46a87" containerName="kserve-container" containerID="cri-o://bdc31b4f3c3e86a8639ab68da9ca8e85e2a6c1f909a0cb72c1905b93a63b433a" gracePeriod=30
Jul 28 17:37:16.297457 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:16.297430    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:37:16.366214 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:16.366183    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:37:17.280091 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:17.280058    2581 generic.go:356] "Generic (PLEG): container finished" podID="c315a841-b6b4-4b79-ab60-2f4658c46a87" containerID="5b5d66374c73638a5c970b2ed559377ad3244ad034e4492392e45743a53c5a24" exitCode=2
Jul 28 17:37:17.280274 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:17.280131    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t" event={"ID":"c315a841-b6b4-4b79-ab60-2f4658c46a87","Type":"ContainerDied","Data":"5b5d66374c73638a5c970b2ed559377ad3244ad034e4492392e45743a53c5a24"}
Jul 28 17:37:17.282095 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:17.282073    2581 generic.go:356] "Generic (PLEG): container finished" podID="ef517b9c-a92a-4148-ab96-64e7dfa3bca4" containerID="0df9c5c54dec281fdece4e6f4dca89ef6f568b6cf75ebbbb54d7ccbc90680d6f" exitCode=2
Jul 28 17:37:17.282203 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:17.282127    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l" event={"ID":"ef517b9c-a92a-4148-ab96-64e7dfa3bca4","Type":"ContainerDied","Data":"0df9c5c54dec281fdece4e6f4dca89ef6f568b6cf75ebbbb54d7ccbc90680d6f"}
Jul 28 17:37:17.976984 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:17.976915    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t" podUID="c315a841-b6b4-4b79-ab60-2f4658c46a87" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 28 17:37:17.981269 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:17.981242    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t" podUID="c315a841-b6b4-4b79-ab60-2f4658c46a87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 17:37:18.980413 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:18.980363    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l" podUID="ef517b9c-a92a-4148-ab96-64e7dfa3bca4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 28 17:37:18.985469 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:18.985441    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l" podUID="ef517b9c-a92a-4148-ab96-64e7dfa3bca4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:37:20.836343 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:20.836320    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"
Jul 28 17:37:20.838856 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:20.838839    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"
Jul 28 17:37:21.019373 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.019294    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c315a841-b6b4-4b79-ab60-2f4658c46a87-proxy-tls\") pod \"c315a841-b6b4-4b79-ab60-2f4658c46a87\" (UID: \"c315a841-b6b4-4b79-ab60-2f4658c46a87\") "
Jul 28 17:37:21.019373 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.019330    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c315a841-b6b4-4b79-ab60-2f4658c46a87-cabundle-cert\") pod \"c315a841-b6b4-4b79-ab60-2f4658c46a87\" (UID: \"c315a841-b6b4-4b79-ab60-2f4658c46a87\") "
Jul 28 17:37:21.019373 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.019350    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-proxy-tls\") pod \"ef517b9c-a92a-4148-ab96-64e7dfa3bca4\" (UID: \"ef517b9c-a92a-4148-ab96-64e7dfa3bca4\") "
Jul 28 17:37:21.019635 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.019394    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c315a841-b6b4-4b79-ab60-2f4658c46a87-kube-rbac-proxy-sar-config\") pod \"c315a841-b6b4-4b79-ab60-2f4658c46a87\" (UID: \"c315a841-b6b4-4b79-ab60-2f4658c46a87\") "
Jul 28 17:37:21.019635 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.019423    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c315a841-b6b4-4b79-ab60-2f4658c46a87-kserve-provision-location\") pod \"c315a841-b6b4-4b79-ab60-2f4658c46a87\" (UID: \"c315a841-b6b4-4b79-ab60-2f4658c46a87\") "
Jul 28 17:37:21.019635 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.019443    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-kube-rbac-proxy-sar-config\") pod \"ef517b9c-a92a-4148-ab96-64e7dfa3bca4\" (UID: \"ef517b9c-a92a-4148-ab96-64e7dfa3bca4\") "
Jul 28 17:37:21.019635 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.019466    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-kserve-provision-location\") pod \"ef517b9c-a92a-4148-ab96-64e7dfa3bca4\" (UID: \"ef517b9c-a92a-4148-ab96-64e7dfa3bca4\") "
Jul 28 17:37:21.019635 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.019545    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-cabundle-cert\") pod \"ef517b9c-a92a-4148-ab96-64e7dfa3bca4\" (UID: \"ef517b9c-a92a-4148-ab96-64e7dfa3bca4\") "
Jul 28 17:37:21.019635 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.019630    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lqp4\" (UniqueName: \"kubernetes.io/projected/c315a841-b6b4-4b79-ab60-2f4658c46a87-kube-api-access-6lqp4\") pod \"c315a841-b6b4-4b79-ab60-2f4658c46a87\" (UID: \"c315a841-b6b4-4b79-ab60-2f4658c46a87\") "
Jul 28 17:37:21.019975 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.019666    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rrz8\" (UniqueName: \"kubernetes.io/projected/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-kube-api-access-7rrz8\") pod \"ef517b9c-a92a-4148-ab96-64e7dfa3bca4\" (UID: \"ef517b9c-a92a-4148-ab96-64e7dfa3bca4\") "
Jul 28 17:37:21.019975 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.019763    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ef517b9c-a92a-4148-ab96-64e7dfa3bca4" (UID: "ef517b9c-a92a-4148-ab96-64e7dfa3bca4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:37:21.019975 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.019783    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c315a841-b6b4-4b79-ab60-2f4658c46a87-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c315a841-b6b4-4b79-ab60-2f4658c46a87" (UID: "c315a841-b6b4-4b79-ab60-2f4658c46a87"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:37:21.019975 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.019810    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c315a841-b6b4-4b79-ab60-2f4658c46a87-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c315a841-b6b4-4b79-ab60-2f4658c46a87" (UID: "c315a841-b6b4-4b79-ab60-2f4658c46a87"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:37:21.019975 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.019815    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ef517b9c-a92a-4148-ab96-64e7dfa3bca4" (UID: "ef517b9c-a92a-4148-ab96-64e7dfa3bca4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:37:21.019975 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.019820    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c315a841-b6b4-4b79-ab60-2f4658c46a87-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c315a841-b6b4-4b79-ab60-2f4658c46a87" (UID: "c315a841-b6b4-4b79-ab60-2f4658c46a87"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:37:21.019975 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.019916    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c315a841-b6b4-4b79-ab60-2f4658c46a87-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:37:21.020295 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.020063    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c315a841-b6b4-4b79-ab60-2f4658c46a87-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:37:21.020295 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.020082    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:37:21.020295 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.020096    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:37:21.020295 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.020109    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c315a841-b6b4-4b79-ab60-2f4658c46a87-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:37:21.020295 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.020184    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ef517b9c-a92a-4148-ab96-64e7dfa3bca4" (UID: "ef517b9c-a92a-4148-ab96-64e7dfa3bca4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:37:21.021530 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.021512    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ef517b9c-a92a-4148-ab96-64e7dfa3bca4" (UID: "ef517b9c-a92a-4148-ab96-64e7dfa3bca4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:37:21.021748 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.021722    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c315a841-b6b4-4b79-ab60-2f4658c46a87-kube-api-access-6lqp4" (OuterVolumeSpecName: "kube-api-access-6lqp4") pod "c315a841-b6b4-4b79-ab60-2f4658c46a87" (UID: "c315a841-b6b4-4b79-ab60-2f4658c46a87"). InnerVolumeSpecName "kube-api-access-6lqp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:37:21.022132 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.022106    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-kube-api-access-7rrz8" (OuterVolumeSpecName: "kube-api-access-7rrz8") pod "ef517b9c-a92a-4148-ab96-64e7dfa3bca4" (UID: "ef517b9c-a92a-4148-ab96-64e7dfa3bca4"). InnerVolumeSpecName "kube-api-access-7rrz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:37:21.022132 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.022118    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c315a841-b6b4-4b79-ab60-2f4658c46a87-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c315a841-b6b4-4b79-ab60-2f4658c46a87" (UID: "c315a841-b6b4-4b79-ab60-2f4658c46a87"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:37:21.120866 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.120826    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:37:21.120866 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.120863    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lqp4\" (UniqueName: \"kubernetes.io/projected/c315a841-b6b4-4b79-ab60-2f4658c46a87-kube-api-access-6lqp4\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:37:21.121054 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.120878    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rrz8\" (UniqueName: \"kubernetes.io/projected/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-kube-api-access-7rrz8\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:37:21.121054 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.120890    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c315a841-b6b4-4b79-ab60-2f4658c46a87-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:37:21.121054 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.120901    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef517b9c-a92a-4148-ab96-64e7dfa3bca4-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:37:21.298272 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.298189    2581 generic.go:356] "Generic (PLEG): container finished" podID="c315a841-b6b4-4b79-ab60-2f4658c46a87" containerID="bdc31b4f3c3e86a8639ab68da9ca8e85e2a6c1f909a0cb72c1905b93a63b433a" exitCode=0
Jul 28 17:37:21.298272 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.298232    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t" event={"ID":"c315a841-b6b4-4b79-ab60-2f4658c46a87","Type":"ContainerDied","Data":"bdc31b4f3c3e86a8639ab68da9ca8e85e2a6c1f909a0cb72c1905b93a63b433a"}
Jul 28 17:37:21.298485 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.298300    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t" event={"ID":"c315a841-b6b4-4b79-ab60-2f4658c46a87","Type":"ContainerDied","Data":"ac25fee2a34af792a04e9aff142fcb4e537ead869db05fcecc490c3d4ed2acc9"}
Jul 28 17:37:21.298485 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.298249    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"
Jul 28 17:37:21.298485 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.298336    2581 scope.go:117] "RemoveContainer" containerID="5b5d66374c73638a5c970b2ed559377ad3244ad034e4492392e45743a53c5a24"
Jul 28 17:37:21.300324 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.300305    2581 generic.go:356] "Generic (PLEG): container finished" podID="ef517b9c-a92a-4148-ab96-64e7dfa3bca4" containerID="2df264421e8de271c2176880030eb833fe0549cb1882c5dda0cfc9ee742bc9df" exitCode=0
Jul 28 17:37:21.300423 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.300377    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"
Jul 28 17:37:21.300486 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.300375    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l" event={"ID":"ef517b9c-a92a-4148-ab96-64e7dfa3bca4","Type":"ContainerDied","Data":"2df264421e8de271c2176880030eb833fe0549cb1882c5dda0cfc9ee742bc9df"}
Jul 28 17:37:21.300547 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.300508    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l" event={"ID":"ef517b9c-a92a-4148-ab96-64e7dfa3bca4","Type":"ContainerDied","Data":"718855b9ba6ec4b2701d95fbec38d0d66556cb5e27d6fcd5584a311fe0e7171a"}
Jul 28 17:37:21.306396 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.306378    2581 scope.go:117] "RemoveContainer" containerID="bdc31b4f3c3e86a8639ab68da9ca8e85e2a6c1f909a0cb72c1905b93a63b433a"
Jul 28 17:37:21.313797 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.313781    2581 scope.go:117] "RemoveContainer" containerID="d3421c8db888f1a5ac9306d0333689d1b82d7524e9fe619321cfcf9435061126"
Jul 28 17:37:21.320682 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.320662    2581 scope.go:117] "RemoveContainer" containerID="5b5d66374c73638a5c970b2ed559377ad3244ad034e4492392e45743a53c5a24"
Jul 28 17:37:21.320933 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:37:21.320915    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b5d66374c73638a5c970b2ed559377ad3244ad034e4492392e45743a53c5a24\": container with ID starting with 5b5d66374c73638a5c970b2ed559377ad3244ad034e4492392e45743a53c5a24 not found: ID does not exist" containerID="5b5d66374c73638a5c970b2ed559377ad3244ad034e4492392e45743a53c5a24"
Jul 28 17:37:21.321048 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.320952    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b5d66374c73638a5c970b2ed559377ad3244ad034e4492392e45743a53c5a24"} err="failed to get container status \"5b5d66374c73638a5c970b2ed559377ad3244ad034e4492392e45743a53c5a24\": rpc error: code = NotFound desc = could not find container \"5b5d66374c73638a5c970b2ed559377ad3244ad034e4492392e45743a53c5a24\": container with ID starting with 5b5d66374c73638a5c970b2ed559377ad3244ad034e4492392e45743a53c5a24 not found: ID does not exist"
Jul 28 17:37:21.321048 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.320970    2581 scope.go:117] "RemoveContainer" containerID="bdc31b4f3c3e86a8639ab68da9ca8e85e2a6c1f909a0cb72c1905b93a63b433a"
Jul 28 17:37:21.321236 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:37:21.321220    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdc31b4f3c3e86a8639ab68da9ca8e85e2a6c1f909a0cb72c1905b93a63b433a\": container with ID starting with bdc31b4f3c3e86a8639ab68da9ca8e85e2a6c1f909a0cb72c1905b93a63b433a not found: ID does not exist" containerID="bdc31b4f3c3e86a8639ab68da9ca8e85e2a6c1f909a0cb72c1905b93a63b433a"
Jul 28 17:37:21.321294 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.321242    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdc31b4f3c3e86a8639ab68da9ca8e85e2a6c1f909a0cb72c1905b93a63b433a"} err="failed to get container status \"bdc31b4f3c3e86a8639ab68da9ca8e85e2a6c1f909a0cb72c1905b93a63b433a\": rpc error: code = NotFound desc = could not find container \"bdc31b4f3c3e86a8639ab68da9ca8e85e2a6c1f909a0cb72c1905b93a63b433a\": container with ID starting with bdc31b4f3c3e86a8639ab68da9ca8e85e2a6c1f909a0cb72c1905b93a63b433a not found: ID does not exist"
Jul 28 17:37:21.321294 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.321256    2581 scope.go:117] "RemoveContainer" containerID="d3421c8db888f1a5ac9306d0333689d1b82d7524e9fe619321cfcf9435061126"
Jul 28 17:37:21.321442 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:37:21.321426    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3421c8db888f1a5ac9306d0333689d1b82d7524e9fe619321cfcf9435061126\": container with ID starting with d3421c8db888f1a5ac9306d0333689d1b82d7524e9fe619321cfcf9435061126 not found: ID does not exist" containerID="d3421c8db888f1a5ac9306d0333689d1b82d7524e9fe619321cfcf9435061126"
Jul 28 17:37:21.321485 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.321444    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3421c8db888f1a5ac9306d0333689d1b82d7524e9fe619321cfcf9435061126"} err="failed to get container status \"d3421c8db888f1a5ac9306d0333689d1b82d7524e9fe619321cfcf9435061126\": rpc error: code = NotFound desc = could not find container \"d3421c8db888f1a5ac9306d0333689d1b82d7524e9fe619321cfcf9435061126\": container with ID starting with d3421c8db888f1a5ac9306d0333689d1b82d7524e9fe619321cfcf9435061126 not found: ID does not exist"
Jul 28 17:37:21.321485 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.321457    2581 scope.go:117] "RemoveContainer" containerID="0df9c5c54dec281fdece4e6f4dca89ef6f568b6cf75ebbbb54d7ccbc90680d6f"
Jul 28 17:37:21.324219 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.324200    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"]
Jul 28 17:37:21.328935 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.328912    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-594f5f9bd5-rlg8t"]
Jul 28 17:37:21.329629 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.329611    2581 scope.go:117] "RemoveContainer" containerID="2df264421e8de271c2176880030eb833fe0549cb1882c5dda0cfc9ee742bc9df"
Jul 28 17:37:21.336440 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.336401    2581 scope.go:117] "RemoveContainer" containerID="f49dd50b37317eaea5ca438b5cc7234c99e5e6b5fd090bf9b7a79b6d4411c034"
Jul 28 17:37:21.343631 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.343608    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"]
Jul 28 17:37:21.344954 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.344916    2581 scope.go:117] "RemoveContainer" containerID="0df9c5c54dec281fdece4e6f4dca89ef6f568b6cf75ebbbb54d7ccbc90680d6f"
Jul 28 17:37:21.345260 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:37:21.345236    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0df9c5c54dec281fdece4e6f4dca89ef6f568b6cf75ebbbb54d7ccbc90680d6f\": container with ID starting with 0df9c5c54dec281fdece4e6f4dca89ef6f568b6cf75ebbbb54d7ccbc90680d6f not found: ID does not exist" containerID="0df9c5c54dec281fdece4e6f4dca89ef6f568b6cf75ebbbb54d7ccbc90680d6f"
Jul 28 17:37:21.345344 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.345265    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0df9c5c54dec281fdece4e6f4dca89ef6f568b6cf75ebbbb54d7ccbc90680d6f"} err="failed to get container status \"0df9c5c54dec281fdece4e6f4dca89ef6f568b6cf75ebbbb54d7ccbc90680d6f\": rpc error: code = NotFound desc = could not find container \"0df9c5c54dec281fdece4e6f4dca89ef6f568b6cf75ebbbb54d7ccbc90680d6f\": container with ID starting with 0df9c5c54dec281fdece4e6f4dca89ef6f568b6cf75ebbbb54d7ccbc90680d6f not found: ID does not exist"
Jul 28 17:37:21.345344 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.345291    2581 scope.go:117] "RemoveContainer" containerID="2df264421e8de271c2176880030eb833fe0549cb1882c5dda0cfc9ee742bc9df"
Jul 28 17:37:21.345550 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:37:21.345530    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2df264421e8de271c2176880030eb833fe0549cb1882c5dda0cfc9ee742bc9df\": container with ID starting with 2df264421e8de271c2176880030eb833fe0549cb1882c5dda0cfc9ee742bc9df not found: ID does not exist" containerID="2df264421e8de271c2176880030eb833fe0549cb1882c5dda0cfc9ee742bc9df"
Jul 28 17:37:21.345611 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.345558    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2df264421e8de271c2176880030eb833fe0549cb1882c5dda0cfc9ee742bc9df"} err="failed to get container status \"2df264421e8de271c2176880030eb833fe0549cb1882c5dda0cfc9ee742bc9df\": rpc error: code = NotFound desc = could not find container \"2df264421e8de271c2176880030eb833fe0549cb1882c5dda0cfc9ee742bc9df\": container with ID starting with 2df264421e8de271c2176880030eb833fe0549cb1882c5dda0cfc9ee742bc9df not found: ID does not exist"
Jul 28 17:37:21.345611 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.345580    2581 scope.go:117] "RemoveContainer" containerID="f49dd50b37317eaea5ca438b5cc7234c99e5e6b5fd090bf9b7a79b6d4411c034"
Jul 28 17:37:21.345849 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:37:21.345830    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f49dd50b37317eaea5ca438b5cc7234c99e5e6b5fd090bf9b7a79b6d4411c034\": container with ID starting with f49dd50b37317eaea5ca438b5cc7234c99e5e6b5fd090bf9b7a79b6d4411c034 not found: ID does not exist" containerID="f49dd50b37317eaea5ca438b5cc7234c99e5e6b5fd090bf9b7a79b6d4411c034"
Jul 28 17:37:21.345910 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.345853    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f49dd50b37317eaea5ca438b5cc7234c99e5e6b5fd090bf9b7a79b6d4411c034"} err="failed to get container status \"f49dd50b37317eaea5ca438b5cc7234c99e5e6b5fd090bf9b7a79b6d4411c034\": rpc error: code = NotFound desc = could not find container \"f49dd50b37317eaea5ca438b5cc7234c99e5e6b5fd090bf9b7a79b6d4411c034\": container with ID starting with f49dd50b37317eaea5ca438b5cc7234c99e5e6b5fd090bf9b7a79b6d4411c034 not found: ID does not exist"
Jul 28 17:37:21.351926 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.351905    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5b976d889b-5rq4l"]
Jul 28 17:37:21.381218 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.381191    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:37:21.450652 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.450626    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"]
Jul 28 17:37:21.450921 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.450902    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c315a841-b6b4-4b79-ab60-2f4658c46a87" containerName="kserve-container"
Jul 28 17:37:21.451024 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.450923    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c315a841-b6b4-4b79-ab60-2f4658c46a87" containerName="kserve-container"
Jul 28 17:37:21.451024 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.450959    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c315a841-b6b4-4b79-ab60-2f4658c46a87" containerName="kube-rbac-proxy"
Jul 28 17:37:21.451024 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.450968    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c315a841-b6b4-4b79-ab60-2f4658c46a87" containerName="kube-rbac-proxy"
Jul 28 17:37:21.451024 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.450981    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c315a841-b6b4-4b79-ab60-2f4658c46a87" containerName="storage-initializer"
Jul 28 17:37:21.451024 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.450990    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c315a841-b6b4-4b79-ab60-2f4658c46a87" containerName="storage-initializer"
Jul 28 17:37:21.451024 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.451001    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef517b9c-a92a-4148-ab96-64e7dfa3bca4" containerName="storage-initializer"
Jul 28 17:37:21.451024 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.451010    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef517b9c-a92a-4148-ab96-64e7dfa3bca4" containerName="storage-initializer"
Jul 28 17:37:21.451024 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.451024    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef517b9c-a92a-4148-ab96-64e7dfa3bca4" containerName="kserve-container"
Jul 28 17:37:21.451368 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.451033    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef517b9c-a92a-4148-ab96-64e7dfa3bca4" containerName="kserve-container"
Jul 28 17:37:21.451368 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.451047    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef517b9c-a92a-4148-ab96-64e7dfa3bca4" containerName="kube-rbac-proxy"
Jul 28 17:37:21.451368 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.451054    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef517b9c-a92a-4148-ab96-64e7dfa3bca4" containerName="kube-rbac-proxy"
Jul 28 17:37:21.451368 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.451126    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef517b9c-a92a-4148-ab96-64e7dfa3bca4" containerName="kube-rbac-proxy"
Jul 28 17:37:21.451368 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.451140    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c315a841-b6b4-4b79-ab60-2f4658c46a87" containerName="kube-rbac-proxy"
Jul 28 17:37:21.451368 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.451149    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef517b9c-a92a-4148-ab96-64e7dfa3bca4" containerName="kserve-container"
Jul 28 17:37:21.451368 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.451158    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c315a841-b6b4-4b79-ab60-2f4658c46a87" containerName="kserve-container"
Jul 28 17:37:21.451368 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.451337    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:37:21.456071 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.456046    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"
Jul 28 17:37:21.457108 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.457084    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"]
Jul 28 17:37:21.458794 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.458773    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:37:21.458888 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.458796    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 28 17:37:21.458888 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.458831    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:37:21.458888 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.458831    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 17:37:21.459385 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.459100    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:37:21.459533 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.459518    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:37:21.459603 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.459518    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Jul 28 17:37:21.460162 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.460148    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"
Jul 28 17:37:21.462667 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.462648    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 28 17:37:21.465158 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.465040    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"]
Jul 28 17:37:21.473174 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.472735    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"]
Jul 28 17:37:21.620564 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.620487    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:37:21.624629 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.624606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh6jh\" (UniqueName: \"kubernetes.io/projected/88b59350-8ec2-439c-91c5-f80ffc0b531a-kube-api-access-nh6jh\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttqwf\" (UID: \"88b59350-8ec2-439c-91c5-f80ffc0b531a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"
Jul 28 17:37:21.624769 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.624643    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/551a7654-5f97-47d6-b016-f2166ab6cef6-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-zz846\" (UID: \"551a7654-5f97-47d6-b016-f2166ab6cef6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"
Jul 28 17:37:21.624769 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.624702    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88b59350-8ec2-439c-91c5-f80ffc0b531a-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttqwf\" (UID: \"88b59350-8ec2-439c-91c5-f80ffc0b531a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"
Jul 28 17:37:21.624769 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.624741    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/551a7654-5f97-47d6-b016-f2166ab6cef6-cabundle-cert\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-zz846\" (UID: \"551a7654-5f97-47d6-b016-f2166ab6cef6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"
Jul 28 17:37:21.624921 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.624774    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88b59350-8ec2-439c-91c5-f80ffc0b531a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttqwf\" (UID: \"88b59350-8ec2-439c-91c5-f80ffc0b531a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"
Jul 28 17:37:21.624921 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.624823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/551a7654-5f97-47d6-b016-f2166ab6cef6-proxy-tls\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-zz846\" (UID: \"551a7654-5f97-47d6-b016-f2166ab6cef6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"
Jul 28 17:37:21.624921 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.624852    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88b59350-8ec2-439c-91c5-f80ffc0b531a-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttqwf\" (UID: \"88b59350-8ec2-439c-91c5-f80ffc0b531a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"
Jul 28 17:37:21.624921 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.624902    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/551a7654-5f97-47d6-b016-f2166ab6cef6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-zz846\" (UID: \"551a7654-5f97-47d6-b016-f2166ab6cef6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"
Jul 28 17:37:21.625089 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.624929    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plz22\" (UniqueName: \"kubernetes.io/projected/551a7654-5f97-47d6-b016-f2166ab6cef6-kube-api-access-plz22\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-zz846\" (UID: \"551a7654-5f97-47d6-b016-f2166ab6cef6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"
Jul 28 17:37:21.625089 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.624976    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88b59350-8ec2-439c-91c5-f80ffc0b531a-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttqwf\" (UID: \"88b59350-8ec2-439c-91c5-f80ffc0b531a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"
Jul 28 17:37:21.717756 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.717727    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:37:21.726184 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.726163    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plz22\" (UniqueName: \"kubernetes.io/projected/551a7654-5f97-47d6-b016-f2166ab6cef6-kube-api-access-plz22\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-zz846\" (UID: \"551a7654-5f97-47d6-b016-f2166ab6cef6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"
Jul 28 17:37:21.726280 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.726194    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88b59350-8ec2-439c-91c5-f80ffc0b531a-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttqwf\" (UID: \"88b59350-8ec2-439c-91c5-f80ffc0b531a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"
Jul 28 17:37:21.726280 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.726214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh6jh\" (UniqueName: \"kubernetes.io/projected/88b59350-8ec2-439c-91c5-f80ffc0b531a-kube-api-access-nh6jh\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttqwf\" (UID: \"88b59350-8ec2-439c-91c5-f80ffc0b531a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"
Jul 28 17:37:21.726369 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.726323    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/551a7654-5f97-47d6-b016-f2166ab6cef6-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-zz846\" (UID: \"551a7654-5f97-47d6-b016-f2166ab6cef6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"
Jul 28 17:37:21.726423 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.726384    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88b59350-8ec2-439c-91c5-f80ffc0b531a-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttqwf\" (UID: \"88b59350-8ec2-439c-91c5-f80ffc0b531a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"
Jul 28 17:37:21.726476 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.726426    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/551a7654-5f97-47d6-b016-f2166ab6cef6-cabundle-cert\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-zz846\" (UID: \"551a7654-5f97-47d6-b016-f2166ab6cef6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"
Jul 28 17:37:21.726476 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.726464    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88b59350-8ec2-439c-91c5-f80ffc0b531a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttqwf\" (UID: \"88b59350-8ec2-439c-91c5-f80ffc0b531a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"
Jul 28 17:37:21.726583 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.726498    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/551a7654-5f97-47d6-b016-f2166ab6cef6-proxy-tls\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-zz846\" (UID: \"551a7654-5f97-47d6-b016-f2166ab6cef6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"
Jul 28 17:37:21.726583 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.726522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88b59350-8ec2-439c-91c5-f80ffc0b531a-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttqwf\" (UID: \"88b59350-8ec2-439c-91c5-f80ffc0b531a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"
Jul 28 17:37:21.726583 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.726529    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88b59350-8ec2-439c-91c5-f80ffc0b531a-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttqwf\" (UID: \"88b59350-8ec2-439c-91c5-f80ffc0b531a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"
Jul 28 17:37:21.726583 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.726557    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/551a7654-5f97-47d6-b016-f2166ab6cef6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-zz846\" (UID: \"551a7654-5f97-47d6-b016-f2166ab6cef6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"
Jul 28 17:37:21.726785 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.726723    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/551a7654-5f97-47d6-b016-f2166ab6cef6-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-zz846\" (UID: \"551a7654-5f97-47d6-b016-f2166ab6cef6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"
Jul 28 17:37:21.727274 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.727233    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/551a7654-5f97-47d6-b016-f2166ab6cef6-cabundle-cert\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-zz846\" (UID: \"551a7654-5f97-47d6-b016-f2166ab6cef6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"
Jul 28 17:37:21.727274 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.727233    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88b59350-8ec2-439c-91c5-f80ffc0b531a-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttqwf\" (UID: \"88b59350-8ec2-439c-91c5-f80ffc0b531a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"
Jul 28 17:37:21.727274 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.727258    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88b59350-8ec2-439c-91c5-f80ffc0b531a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttqwf\" (UID: \"88b59350-8ec2-439c-91c5-f80ffc0b531a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"
Jul 28 17:37:21.727426 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.727258    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/551a7654-5f97-47d6-b016-f2166ab6cef6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-zz846\" (UID: \"551a7654-5f97-47d6-b016-f2166ab6cef6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"
Jul 28 17:37:21.728910 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.728893    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88b59350-8ec2-439c-91c5-f80ffc0b531a-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttqwf\" (UID: \"88b59350-8ec2-439c-91c5-f80ffc0b531a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"
Jul 28 17:37:21.728970 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.728895    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/551a7654-5f97-47d6-b016-f2166ab6cef6-proxy-tls\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-zz846\" (UID: \"551a7654-5f97-47d6-b016-f2166ab6cef6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"
Jul 28 17:37:21.735000 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.734979    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh6jh\" (UniqueName: \"kubernetes.io/projected/88b59350-8ec2-439c-91c5-f80ffc0b531a-kube-api-access-nh6jh\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-ttqwf\" (UID: \"88b59350-8ec2-439c-91c5-f80ffc0b531a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"
Jul 28 17:37:21.735204 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.735186    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plz22\" (UniqueName: \"kubernetes.io/projected/551a7654-5f97-47d6-b016-f2166ab6cef6-kube-api-access-plz22\") pod \"isvc-canary-promote-predictor-5d7dcd97d5-zz846\" (UID: \"551a7654-5f97-47d6-b016-f2166ab6cef6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"
Jul 28 17:37:21.770093 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.770065    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"
Jul 28 17:37:21.777321 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.777299    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"
Jul 28 17:37:21.903852 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.903793    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"]
Jul 28 17:37:21.906318 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:37:21.906290    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551a7654_5f97_47d6_b016_f2166ab6cef6.slice/crio-0ff898a2d64abb95d27bf61c4426a160d5dd45c73e159356cfc80319e87d91c2 WatchSource:0}: Error finding container 0ff898a2d64abb95d27bf61c4426a160d5dd45c73e159356cfc80319e87d91c2: Status 404 returned error can't find the container with id 0ff898a2d64abb95d27bf61c4426a160d5dd45c73e159356cfc80319e87d91c2
Jul 28 17:37:21.925357 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:21.925301    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"]
Jul 28 17:37:21.940973 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:37:21.940925    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88b59350_8ec2_439c_91c5_f80ffc0b531a.slice/crio-adbb8d6da4e3cfee465865ba689d1698903534f89cc8d3f599cb0df26486fa18 WatchSource:0}: Error finding container adbb8d6da4e3cfee465865ba689d1698903534f89cc8d3f599cb0df26486fa18: Status 404 returned error can't find the container with id adbb8d6da4e3cfee465865ba689d1698903534f89cc8d3f599cb0df26486fa18
Jul 28 17:37:22.290512 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:22.290477    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c315a841-b6b4-4b79-ab60-2f4658c46a87" path="/var/lib/kubelet/pods/c315a841-b6b4-4b79-ab60-2f4658c46a87/volumes"
Jul 28 17:37:22.290960 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:22.290932    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef517b9c-a92a-4148-ab96-64e7dfa3bca4" path="/var/lib/kubelet/pods/ef517b9c-a92a-4148-ab96-64e7dfa3bca4/volumes"
Jul 28 17:37:22.306422 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:22.306386    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846" event={"ID":"551a7654-5f97-47d6-b016-f2166ab6cef6","Type":"ContainerStarted","Data":"26e3fb60535ff81a173ea887e8e1cf3af2c588a372a2d0483af82613641b4c9c"}
Jul 28 17:37:22.306588 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:22.306455    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846" event={"ID":"551a7654-5f97-47d6-b016-f2166ab6cef6","Type":"ContainerStarted","Data":"0ff898a2d64abb95d27bf61c4426a160d5dd45c73e159356cfc80319e87d91c2"}
Jul 28 17:37:22.308837 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:22.308804    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf" event={"ID":"88b59350-8ec2-439c-91c5-f80ffc0b531a","Type":"ContainerStarted","Data":"1f0df026b1462427ed01612a3ea6b3066f4254b7c0eb4a1fece5be9e9c93eff9"}
Jul 28 17:37:22.308954 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:22.308867    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf" event={"ID":"88b59350-8ec2-439c-91c5-f80ffc0b531a","Type":"ContainerStarted","Data":"adbb8d6da4e3cfee465865ba689d1698903534f89cc8d3f599cb0df26486fa18"}
Jul 28 17:37:26.323062 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:26.322978    2581 generic.go:356] "Generic (PLEG): container finished" podID="551a7654-5f97-47d6-b016-f2166ab6cef6" containerID="26e3fb60535ff81a173ea887e8e1cf3af2c588a372a2d0483af82613641b4c9c" exitCode=0
Jul 28 17:37:26.323471 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:26.323051    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846" event={"ID":"551a7654-5f97-47d6-b016-f2166ab6cef6","Type":"ContainerDied","Data":"26e3fb60535ff81a173ea887e8e1cf3af2c588a372a2d0483af82613641b4c9c"}
Jul 28 17:37:26.324676 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:26.324653    2581 generic.go:356] "Generic (PLEG): container finished" podID="88b59350-8ec2-439c-91c5-f80ffc0b531a" containerID="1f0df026b1462427ed01612a3ea6b3066f4254b7c0eb4a1fece5be9e9c93eff9" exitCode=0
Jul 28 17:37:26.324769 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:26.324701    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf" event={"ID":"88b59350-8ec2-439c-91c5-f80ffc0b531a","Type":"ContainerDied","Data":"1f0df026b1462427ed01612a3ea6b3066f4254b7c0eb4a1fece5be9e9c93eff9"}
Jul 28 17:37:27.330957 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:27.330902    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf" event={"ID":"88b59350-8ec2-439c-91c5-f80ffc0b531a","Type":"ContainerStarted","Data":"9ebf7bb44635d3b7c0be3f3d4d8897a6a1ef5518af4ec593190ce39473a59189"}
Jul 28 17:37:27.331362 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:27.331020    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf" event={"ID":"88b59350-8ec2-439c-91c5-f80ffc0b531a","Type":"ContainerStarted","Data":"6bb826e245d2596672cb0b379d562f6b2e85b44ee84c6992e84b0179b6415521"}
Jul 28 17:37:27.331362 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:27.331298    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"
Jul 28 17:37:27.331500 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:27.331366    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"
Jul 28 17:37:27.332776 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:27.332741    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf" podUID="88b59350-8ec2-439c-91c5-f80ffc0b531a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 17:37:27.333059 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:27.333037    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846" event={"ID":"551a7654-5f97-47d6-b016-f2166ab6cef6","Type":"ContainerStarted","Data":"acb918dc1d2df72966be447c9608ae91e2b21a6fb080f3449bf0ce5c657cce52"}
Jul 28 17:37:27.333164 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:27.333093    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846" event={"ID":"551a7654-5f97-47d6-b016-f2166ab6cef6","Type":"ContainerStarted","Data":"65267b2feeca789a4eee8d3e6905efcaa39d48a7e1c5af8b984d3d03129fee3b"}
Jul 28 17:37:27.333302 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:27.333286    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"
Jul 28 17:37:27.353055 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:27.352970    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf" podStartSLOduration=2.377437061 podStartE2EDuration="6.352958372s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.975521311s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:37:27.351549513 +0000 UTC m=+1695.626165950" watchObservedRunningTime="2026-07-28 17:37:27.352958372 +0000 UTC m=+1695.627574797"
Jul 28 17:37:27.370757 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:27.370714    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846" podStartSLOduration=2.641942498 podStartE2EDuration="6.370697686s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.728755188s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:37:27.369563192 +0000 UTC m=+1695.644179634" watchObservedRunningTime="2026-07-28 17:37:27.370697686 +0000 UTC m=+1695.645314119"
Jul 28 17:37:28.337684 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:28.337647    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf" podUID="88b59350-8ec2-439c-91c5-f80ffc0b531a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 17:37:28.338150 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:28.337716    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"
Jul 28 17:37:28.338751 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:28.338729    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846" podUID="551a7654-5f97-47d6-b016-f2166ab6cef6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:37:29.341129 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:29.341092    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846" podUID="551a7654-5f97-47d6-b016-f2166ab6cef6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:37:33.342674 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:33.342642    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"
Jul 28 17:37:33.343204 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:33.343173    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf" podUID="88b59350-8ec2-439c-91c5-f80ffc0b531a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 17:37:34.345674 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:34.345647    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"
Jul 28 17:37:34.346085 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:34.346060    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846" podUID="551a7654-5f97-47d6-b016-f2166ab6cef6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:37:43.343777 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:43.343697    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf" podUID="88b59350-8ec2-439c-91c5-f80ffc0b531a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 17:37:44.347095 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:44.347053    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846" podUID="551a7654-5f97-47d6-b016-f2166ab6cef6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:37:53.343899 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:53.343860    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf" podUID="88b59350-8ec2-439c-91c5-f80ffc0b531a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 17:37:54.346151 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:37:54.346108    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846" podUID="551a7654-5f97-47d6-b016-f2166ab6cef6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:38:03.343907 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:03.343857    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf" podUID="88b59350-8ec2-439c-91c5-f80ffc0b531a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 17:38:04.346754 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:04.346718    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846" podUID="551a7654-5f97-47d6-b016-f2166ab6cef6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:38:13.343177 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:13.343140    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf" podUID="88b59350-8ec2-439c-91c5-f80ffc0b531a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 17:38:14.346505 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:14.346462    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846" podUID="551a7654-5f97-47d6-b016-f2166ab6cef6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:38:23.343136 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:23.343094    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf" podUID="88b59350-8ec2-439c-91c5-f80ffc0b531a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 17:38:24.346497 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:24.346463    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846" podUID="551a7654-5f97-47d6-b016-f2166ab6cef6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:38:33.343469 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:33.343430    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf" podUID="88b59350-8ec2-439c-91c5-f80ffc0b531a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 17:38:34.346886 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:34.346849    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846" podUID="551a7654-5f97-47d6-b016-f2166ab6cef6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:38:43.344469 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:43.344433    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"
Jul 28 17:38:44.346640 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:44.346605    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"
Jul 28 17:38:51.524044 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:51.524012    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"]
Jul 28 17:38:51.524431 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:51.524282    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846" podUID="551a7654-5f97-47d6-b016-f2166ab6cef6" containerName="kserve-container" containerID="cri-o://65267b2feeca789a4eee8d3e6905efcaa39d48a7e1c5af8b984d3d03129fee3b" gracePeriod=30
Jul 28 17:38:51.524431 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:51.524328    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846" podUID="551a7654-5f97-47d6-b016-f2166ab6cef6" containerName="kube-rbac-proxy" containerID="cri-o://acb918dc1d2df72966be447c9608ae91e2b21a6fb080f3449bf0ce5c657cce52" gracePeriod=30
Jul 28 17:38:51.547321 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:51.547295    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:51.804059 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:51.803929    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:38:52.637572 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:52.637539    2581 generic.go:356] "Generic (PLEG): container finished" podID="551a7654-5f97-47d6-b016-f2166ab6cef6" containerID="acb918dc1d2df72966be447c9608ae91e2b21a6fb080f3449bf0ce5c657cce52" exitCode=2
Jul 28 17:38:52.637972 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:52.637616    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846" event={"ID":"551a7654-5f97-47d6-b016-f2166ab6cef6","Type":"ContainerDied","Data":"acb918dc1d2df72966be447c9608ae91e2b21a6fb080f3449bf0ce5c657cce52"}
Jul 28 17:38:54.342286 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:54.342252    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846" podUID="551a7654-5f97-47d6-b016-f2166ab6cef6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 28 17:38:54.346721 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:54.346699    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846" podUID="551a7654-5f97-47d6-b016-f2166ab6cef6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:38:55.959995 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:55.959970    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"
Jul 28 17:38:56.011732 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.011705    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/551a7654-5f97-47d6-b016-f2166ab6cef6-kube-rbac-proxy-sar-config\") pod \"551a7654-5f97-47d6-b016-f2166ab6cef6\" (UID: \"551a7654-5f97-47d6-b016-f2166ab6cef6\") "
Jul 28 17:38:56.011860 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.011745    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plz22\" (UniqueName: \"kubernetes.io/projected/551a7654-5f97-47d6-b016-f2166ab6cef6-kube-api-access-plz22\") pod \"551a7654-5f97-47d6-b016-f2166ab6cef6\" (UID: \"551a7654-5f97-47d6-b016-f2166ab6cef6\") "
Jul 28 17:38:56.011860 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.011780    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/551a7654-5f97-47d6-b016-f2166ab6cef6-proxy-tls\") pod \"551a7654-5f97-47d6-b016-f2166ab6cef6\" (UID: \"551a7654-5f97-47d6-b016-f2166ab6cef6\") "
Jul 28 17:38:56.012003 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.011900    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/551a7654-5f97-47d6-b016-f2166ab6cef6-kserve-provision-location\") pod \"551a7654-5f97-47d6-b016-f2166ab6cef6\" (UID: \"551a7654-5f97-47d6-b016-f2166ab6cef6\") "
Jul 28 17:38:56.012063 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.012012    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/551a7654-5f97-47d6-b016-f2166ab6cef6-cabundle-cert\") pod \"551a7654-5f97-47d6-b016-f2166ab6cef6\" (UID: \"551a7654-5f97-47d6-b016-f2166ab6cef6\") "
Jul 28 17:38:56.012193 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.012167    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551a7654-5f97-47d6-b016-f2166ab6cef6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "551a7654-5f97-47d6-b016-f2166ab6cef6" (UID: "551a7654-5f97-47d6-b016-f2166ab6cef6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:38:56.012193 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.012177    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551a7654-5f97-47d6-b016-f2166ab6cef6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "551a7654-5f97-47d6-b016-f2166ab6cef6" (UID: "551a7654-5f97-47d6-b016-f2166ab6cef6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:38:56.012406 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.012385    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551a7654-5f97-47d6-b016-f2166ab6cef6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "551a7654-5f97-47d6-b016-f2166ab6cef6" (UID: "551a7654-5f97-47d6-b016-f2166ab6cef6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:38:56.013768 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.013750    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/551a7654-5f97-47d6-b016-f2166ab6cef6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "551a7654-5f97-47d6-b016-f2166ab6cef6" (UID: "551a7654-5f97-47d6-b016-f2166ab6cef6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:38:56.013768 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.013758    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551a7654-5f97-47d6-b016-f2166ab6cef6-kube-api-access-plz22" (OuterVolumeSpecName: "kube-api-access-plz22") pod "551a7654-5f97-47d6-b016-f2166ab6cef6" (UID: "551a7654-5f97-47d6-b016-f2166ab6cef6"). InnerVolumeSpecName "kube-api-access-plz22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:38:56.112449 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.112425    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/551a7654-5f97-47d6-b016-f2166ab6cef6-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:38:56.112449 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.112445    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/551a7654-5f97-47d6-b016-f2166ab6cef6-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:38:56.112590 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.112455    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/551a7654-5f97-47d6-b016-f2166ab6cef6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:38:56.112590 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.112463    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plz22\" (UniqueName: \"kubernetes.io/projected/551a7654-5f97-47d6-b016-f2166ab6cef6-kube-api-access-plz22\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:38:56.112590 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.112472    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/551a7654-5f97-47d6-b016-f2166ab6cef6-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:38:56.653115 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.653081    2581 generic.go:356] "Generic (PLEG): container finished" podID="551a7654-5f97-47d6-b016-f2166ab6cef6" containerID="65267b2feeca789a4eee8d3e6905efcaa39d48a7e1c5af8b984d3d03129fee3b" exitCode=0
Jul 28 17:38:56.653270 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.653152    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"
Jul 28 17:38:56.653270 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.653163    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846" event={"ID":"551a7654-5f97-47d6-b016-f2166ab6cef6","Type":"ContainerDied","Data":"65267b2feeca789a4eee8d3e6905efcaa39d48a7e1c5af8b984d3d03129fee3b"}
Jul 28 17:38:56.653270 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.653226    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846" event={"ID":"551a7654-5f97-47d6-b016-f2166ab6cef6","Type":"ContainerDied","Data":"0ff898a2d64abb95d27bf61c4426a160d5dd45c73e159356cfc80319e87d91c2"}
Jul 28 17:38:56.653270 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.653254    2581 scope.go:117] "RemoveContainer" containerID="acb918dc1d2df72966be447c9608ae91e2b21a6fb080f3449bf0ce5c657cce52"
Jul 28 17:38:56.660971 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.660955    2581 scope.go:117] "RemoveContainer" containerID="65267b2feeca789a4eee8d3e6905efcaa39d48a7e1c5af8b984d3d03129fee3b"
Jul 28 17:38:56.667597 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.667581    2581 scope.go:117] "RemoveContainer" containerID="26e3fb60535ff81a173ea887e8e1cf3af2c588a372a2d0483af82613641b4c9c"
Jul 28 17:38:56.671859 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.671838    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"]
Jul 28 17:38:56.674392 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.674375    2581 scope.go:117] "RemoveContainer" containerID="acb918dc1d2df72966be447c9608ae91e2b21a6fb080f3449bf0ce5c657cce52"
Jul 28 17:38:56.674666 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:38:56.674639    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acb918dc1d2df72966be447c9608ae91e2b21a6fb080f3449bf0ce5c657cce52\": container with ID starting with acb918dc1d2df72966be447c9608ae91e2b21a6fb080f3449bf0ce5c657cce52 not found: ID does not exist" containerID="acb918dc1d2df72966be447c9608ae91e2b21a6fb080f3449bf0ce5c657cce52"
Jul 28 17:38:56.674715 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.674677    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acb918dc1d2df72966be447c9608ae91e2b21a6fb080f3449bf0ce5c657cce52"} err="failed to get container status \"acb918dc1d2df72966be447c9608ae91e2b21a6fb080f3449bf0ce5c657cce52\": rpc error: code = NotFound desc = could not find container \"acb918dc1d2df72966be447c9608ae91e2b21a6fb080f3449bf0ce5c657cce52\": container with ID starting with acb918dc1d2df72966be447c9608ae91e2b21a6fb080f3449bf0ce5c657cce52 not found: ID does not exist"
Jul 28 17:38:56.674715 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.674694    2581 scope.go:117] "RemoveContainer" containerID="65267b2feeca789a4eee8d3e6905efcaa39d48a7e1c5af8b984d3d03129fee3b"
Jul 28 17:38:56.674923 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:38:56.674906    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65267b2feeca789a4eee8d3e6905efcaa39d48a7e1c5af8b984d3d03129fee3b\": container with ID starting with 65267b2feeca789a4eee8d3e6905efcaa39d48a7e1c5af8b984d3d03129fee3b not found: ID does not exist" containerID="65267b2feeca789a4eee8d3e6905efcaa39d48a7e1c5af8b984d3d03129fee3b"
Jul 28 17:38:56.674991 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.674930    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65267b2feeca789a4eee8d3e6905efcaa39d48a7e1c5af8b984d3d03129fee3b"} err="failed to get container status \"65267b2feeca789a4eee8d3e6905efcaa39d48a7e1c5af8b984d3d03129fee3b\": rpc error: code = NotFound desc = could not find container \"65267b2feeca789a4eee8d3e6905efcaa39d48a7e1c5af8b984d3d03129fee3b\": container with ID starting with 65267b2feeca789a4eee8d3e6905efcaa39d48a7e1c5af8b984d3d03129fee3b not found: ID does not exist"
Jul 28 17:38:56.674991 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.674959    2581 scope.go:117] "RemoveContainer" containerID="26e3fb60535ff81a173ea887e8e1cf3af2c588a372a2d0483af82613641b4c9c"
Jul 28 17:38:56.675232 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:38:56.675207    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26e3fb60535ff81a173ea887e8e1cf3af2c588a372a2d0483af82613641b4c9c\": container with ID starting with 26e3fb60535ff81a173ea887e8e1cf3af2c588a372a2d0483af82613641b4c9c not found: ID does not exist" containerID="26e3fb60535ff81a173ea887e8e1cf3af2c588a372a2d0483af82613641b4c9c"
Jul 28 17:38:56.675340 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.675237    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26e3fb60535ff81a173ea887e8e1cf3af2c588a372a2d0483af82613641b4c9c"} err="failed to get container status \"26e3fb60535ff81a173ea887e8e1cf3af2c588a372a2d0483af82613641b4c9c\": rpc error: code = NotFound desc = could not find container \"26e3fb60535ff81a173ea887e8e1cf3af2c588a372a2d0483af82613641b4c9c\": container with ID starting with 26e3fb60535ff81a173ea887e8e1cf3af2c588a372a2d0483af82613641b4c9c not found: ID does not exist"
Jul 28 17:38:56.676797 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:56.676778    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5d7dcd97d5-zz846"]
Jul 28 17:38:58.289435 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:38:58.289403    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="551a7654-5f97-47d6-b016-f2166ab6cef6" path="/var/lib/kubelet/pods/551a7654-5f97-47d6-b016-f2166ab6cef6/volumes"
Jul 28 17:39:12.276163 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:39:12.276136    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 17:39:12.277907 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:39:12.277888    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 17:40:31.762824 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:31.762789    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"]
Jul 28 17:40:31.765320 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:31.763046    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf" podUID="88b59350-8ec2-439c-91c5-f80ffc0b531a" containerName="kserve-container" containerID="cri-o://6bb826e245d2596672cb0b379d562f6b2e85b44ee84c6992e84b0179b6415521" gracePeriod=30
Jul 28 17:40:31.765320 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:31.763071    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf" podUID="88b59350-8ec2-439c-91c5-f80ffc0b531a" containerName="kube-rbac-proxy" containerID="cri-o://9ebf7bb44635d3b7c0be3f3d4d8897a6a1ef5518af4ec593190ce39473a59189" gracePeriod=30
Jul 28 17:40:31.822910 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:31.822883    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:40:31.975245 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:31.975217    2581 generic.go:356] "Generic (PLEG): container finished" podID="88b59350-8ec2-439c-91c5-f80ffc0b531a" containerID="9ebf7bb44635d3b7c0be3f3d4d8897a6a1ef5518af4ec593190ce39473a59189" exitCode=2
Jul 28 17:40:31.975448 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:31.975289    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf" event={"ID":"88b59350-8ec2-439c-91c5-f80ffc0b531a","Type":"ContainerDied","Data":"9ebf7bb44635d3b7c0be3f3d4d8897a6a1ef5518af4ec593190ce39473a59189"}
Jul 28 17:40:33.338707 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:33.338669    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf" podUID="88b59350-8ec2-439c-91c5-f80ffc0b531a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 28 17:40:33.344023 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:33.344003    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf" podUID="88b59350-8ec2-439c-91c5-f80ffc0b531a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 17:40:35.916908 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:35.916887    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"
Jul 28 17:40:35.991876 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:35.991846    2581 generic.go:356] "Generic (PLEG): container finished" podID="88b59350-8ec2-439c-91c5-f80ffc0b531a" containerID="6bb826e245d2596672cb0b379d562f6b2e85b44ee84c6992e84b0179b6415521" exitCode=0
Jul 28 17:40:35.992043 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:35.991898    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"
Jul 28 17:40:35.992043 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:35.991900    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf" event={"ID":"88b59350-8ec2-439c-91c5-f80ffc0b531a","Type":"ContainerDied","Data":"6bb826e245d2596672cb0b379d562f6b2e85b44ee84c6992e84b0179b6415521"}
Jul 28 17:40:35.992043 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:35.991996    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf" event={"ID":"88b59350-8ec2-439c-91c5-f80ffc0b531a","Type":"ContainerDied","Data":"adbb8d6da4e3cfee465865ba689d1698903534f89cc8d3f599cb0df26486fa18"}
Jul 28 17:40:35.992043 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:35.992023    2581 scope.go:117] "RemoveContainer" containerID="9ebf7bb44635d3b7c0be3f3d4d8897a6a1ef5518af4ec593190ce39473a59189"
Jul 28 17:40:35.999470 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:35.999453    2581 scope.go:117] "RemoveContainer" containerID="6bb826e245d2596672cb0b379d562f6b2e85b44ee84c6992e84b0179b6415521"
Jul 28 17:40:36.006308 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.006292    2581 scope.go:117] "RemoveContainer" containerID="1f0df026b1462427ed01612a3ea6b3066f4254b7c0eb4a1fece5be9e9c93eff9"
Jul 28 17:40:36.012394 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.012376    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88b59350-8ec2-439c-91c5-f80ffc0b531a-kube-rbac-proxy-sar-config\") pod \"88b59350-8ec2-439c-91c5-f80ffc0b531a\" (UID: \"88b59350-8ec2-439c-91c5-f80ffc0b531a\") "
Jul 28 17:40:36.012487 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.012424    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88b59350-8ec2-439c-91c5-f80ffc0b531a-cabundle-cert\") pod \"88b59350-8ec2-439c-91c5-f80ffc0b531a\" (UID: \"88b59350-8ec2-439c-91c5-f80ffc0b531a\") "
Jul 28 17:40:36.012487 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.012455    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nh6jh\" (UniqueName: \"kubernetes.io/projected/88b59350-8ec2-439c-91c5-f80ffc0b531a-kube-api-access-nh6jh\") pod \"88b59350-8ec2-439c-91c5-f80ffc0b531a\" (UID: \"88b59350-8ec2-439c-91c5-f80ffc0b531a\") "
Jul 28 17:40:36.012487 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.012473    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88b59350-8ec2-439c-91c5-f80ffc0b531a-proxy-tls\") pod \"88b59350-8ec2-439c-91c5-f80ffc0b531a\" (UID: \"88b59350-8ec2-439c-91c5-f80ffc0b531a\") "
Jul 28 17:40:36.012756 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.012497    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88b59350-8ec2-439c-91c5-f80ffc0b531a-kserve-provision-location\") pod \"88b59350-8ec2-439c-91c5-f80ffc0b531a\" (UID: \"88b59350-8ec2-439c-91c5-f80ffc0b531a\") "
Jul 28 17:40:36.012817 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.012796    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88b59350-8ec2-439c-91c5-f80ffc0b531a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "88b59350-8ec2-439c-91c5-f80ffc0b531a" (UID: "88b59350-8ec2-439c-91c5-f80ffc0b531a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:40:36.012881 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.012828    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88b59350-8ec2-439c-91c5-f80ffc0b531a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "88b59350-8ec2-439c-91c5-f80ffc0b531a" (UID: "88b59350-8ec2-439c-91c5-f80ffc0b531a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:40:36.012935 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.012866    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88b59350-8ec2-439c-91c5-f80ffc0b531a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "88b59350-8ec2-439c-91c5-f80ffc0b531a" (UID: "88b59350-8ec2-439c-91c5-f80ffc0b531a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:40:36.013241 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.013221    2581 scope.go:117] "RemoveContainer" containerID="9ebf7bb44635d3b7c0be3f3d4d8897a6a1ef5518af4ec593190ce39473a59189"
Jul 28 17:40:36.013517 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:40:36.013490    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ebf7bb44635d3b7c0be3f3d4d8897a6a1ef5518af4ec593190ce39473a59189\": container with ID starting with 9ebf7bb44635d3b7c0be3f3d4d8897a6a1ef5518af4ec593190ce39473a59189 not found: ID does not exist" containerID="9ebf7bb44635d3b7c0be3f3d4d8897a6a1ef5518af4ec593190ce39473a59189"
Jul 28 17:40:36.013583 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.013519    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ebf7bb44635d3b7c0be3f3d4d8897a6a1ef5518af4ec593190ce39473a59189"} err="failed to get container status \"9ebf7bb44635d3b7c0be3f3d4d8897a6a1ef5518af4ec593190ce39473a59189\": rpc error: code = NotFound desc = could not find container \"9ebf7bb44635d3b7c0be3f3d4d8897a6a1ef5518af4ec593190ce39473a59189\": container with ID starting with 9ebf7bb44635d3b7c0be3f3d4d8897a6a1ef5518af4ec593190ce39473a59189 not found: ID does not exist"
Jul 28 17:40:36.013583 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.013539    2581 scope.go:117] "RemoveContainer" containerID="6bb826e245d2596672cb0b379d562f6b2e85b44ee84c6992e84b0179b6415521"
Jul 28 17:40:36.013795 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:40:36.013779    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bb826e245d2596672cb0b379d562f6b2e85b44ee84c6992e84b0179b6415521\": container with ID starting with 6bb826e245d2596672cb0b379d562f6b2e85b44ee84c6992e84b0179b6415521 not found: ID does not exist" containerID="6bb826e245d2596672cb0b379d562f6b2e85b44ee84c6992e84b0179b6415521"
Jul 28 17:40:36.013853 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.013800    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb826e245d2596672cb0b379d562f6b2e85b44ee84c6992e84b0179b6415521"} err="failed to get container status \"6bb826e245d2596672cb0b379d562f6b2e85b44ee84c6992e84b0179b6415521\": rpc error: code = NotFound desc = could not find container \"6bb826e245d2596672cb0b379d562f6b2e85b44ee84c6992e84b0179b6415521\": container with ID starting with 6bb826e245d2596672cb0b379d562f6b2e85b44ee84c6992e84b0179b6415521 not found: ID does not exist"
Jul 28 17:40:36.013853 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.013815    2581 scope.go:117] "RemoveContainer" containerID="1f0df026b1462427ed01612a3ea6b3066f4254b7c0eb4a1fece5be9e9c93eff9"
Jul 28 17:40:36.014127 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:40:36.014089    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f0df026b1462427ed01612a3ea6b3066f4254b7c0eb4a1fece5be9e9c93eff9\": container with ID starting with 1f0df026b1462427ed01612a3ea6b3066f4254b7c0eb4a1fece5be9e9c93eff9 not found: ID does not exist" containerID="1f0df026b1462427ed01612a3ea6b3066f4254b7c0eb4a1fece5be9e9c93eff9"
Jul 28 17:40:36.014127 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.014111    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f0df026b1462427ed01612a3ea6b3066f4254b7c0eb4a1fece5be9e9c93eff9"} err="failed to get container status \"1f0df026b1462427ed01612a3ea6b3066f4254b7c0eb4a1fece5be9e9c93eff9\": rpc error: code = NotFound desc = could not find container \"1f0df026b1462427ed01612a3ea6b3066f4254b7c0eb4a1fece5be9e9c93eff9\": container with ID starting with 1f0df026b1462427ed01612a3ea6b3066f4254b7c0eb4a1fece5be9e9c93eff9 not found: ID does not exist"
Jul 28 17:40:36.014527 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.014506    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88b59350-8ec2-439c-91c5-f80ffc0b531a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "88b59350-8ec2-439c-91c5-f80ffc0b531a" (UID: "88b59350-8ec2-439c-91c5-f80ffc0b531a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:40:36.014587 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.014552    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b59350-8ec2-439c-91c5-f80ffc0b531a-kube-api-access-nh6jh" (OuterVolumeSpecName: "kube-api-access-nh6jh") pod "88b59350-8ec2-439c-91c5-f80ffc0b531a" (UID: "88b59350-8ec2-439c-91c5-f80ffc0b531a"). InnerVolumeSpecName "kube-api-access-nh6jh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:40:36.113869 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.113832    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88b59350-8ec2-439c-91c5-f80ffc0b531a-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:40:36.113869 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.113866    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh6jh\" (UniqueName: \"kubernetes.io/projected/88b59350-8ec2-439c-91c5-f80ffc0b531a-kube-api-access-nh6jh\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:40:36.113869 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.113878    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88b59350-8ec2-439c-91c5-f80ffc0b531a-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:40:36.114128 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.113886    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88b59350-8ec2-439c-91c5-f80ffc0b531a-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:40:36.114128 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.113894    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88b59350-8ec2-439c-91c5-f80ffc0b531a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:40:36.314465 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.314432    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"]
Jul 28 17:40:36.317839 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.317817    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-ttqwf"]
Jul 28 17:40:36.875367 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.875328    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:40:36.944321 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.944275    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:40:36.947991 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.947963    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"]
Jul 28 17:40:36.948233 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.948221    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88b59350-8ec2-439c-91c5-f80ffc0b531a" containerName="storage-initializer"
Jul 28 17:40:36.948276 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.948235    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="88b59350-8ec2-439c-91c5-f80ffc0b531a" containerName="storage-initializer"
Jul 28 17:40:36.948276 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.948244    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="551a7654-5f97-47d6-b016-f2166ab6cef6" containerName="storage-initializer"
Jul 28 17:40:36.948276 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.948249    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="551a7654-5f97-47d6-b016-f2166ab6cef6" containerName="storage-initializer"
Jul 28 17:40:36.948276 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.948258    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="551a7654-5f97-47d6-b016-f2166ab6cef6" containerName="kube-rbac-proxy"
Jul 28 17:40:36.948276 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.948264    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="551a7654-5f97-47d6-b016-f2166ab6cef6" containerName="kube-rbac-proxy"
Jul 28 17:40:36.948276 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.948271    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88b59350-8ec2-439c-91c5-f80ffc0b531a" containerName="kserve-container"
Jul 28 17:40:36.948276 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.948276    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="88b59350-8ec2-439c-91c5-f80ffc0b531a" containerName="kserve-container"
Jul 28 17:40:36.948495 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.948285    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="551a7654-5f97-47d6-b016-f2166ab6cef6" containerName="kserve-container"
Jul 28 17:40:36.948495 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.948291    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="551a7654-5f97-47d6-b016-f2166ab6cef6" containerName="kserve-container"
Jul 28 17:40:36.948495 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.948297    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88b59350-8ec2-439c-91c5-f80ffc0b531a" containerName="kube-rbac-proxy"
Jul 28 17:40:36.948495 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.948309    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="88b59350-8ec2-439c-91c5-f80ffc0b531a" containerName="kube-rbac-proxy"
Jul 28 17:40:36.948495 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.948352    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="88b59350-8ec2-439c-91c5-f80ffc0b531a" containerName="kserve-container"
Jul 28 17:40:36.948495 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.948359    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="88b59350-8ec2-439c-91c5-f80ffc0b531a" containerName="kube-rbac-proxy"
Jul 28 17:40:36.948495 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.948364    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="551a7654-5f97-47d6-b016-f2166ab6cef6" containerName="kube-rbac-proxy"
Jul 28 17:40:36.948495 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.948370    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="551a7654-5f97-47d6-b016-f2166ab6cef6" containerName="kserve-container"
Jul 28 17:40:36.952807 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.952787    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"
Jul 28 17:40:36.955444 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.955415    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 28 17:40:36.955444 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.955432    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:40:36.955614 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.955470    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 28 17:40:36.955739 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.955720    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:40:36.955858 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.955781    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:40:36.956898 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.956876    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 17:40:36.957014 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.956876    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:40:36.959104 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.959058    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"]
Jul 28 17:40:36.962433 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.962415    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"
Jul 28 17:40:36.965243 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.965223    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Jul 28 17:40:36.966785 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.966762    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"]
Jul 28 17:40:36.973617 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:36.973599    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"]
Jul 28 17:40:37.020081 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.020055    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddb77b27-527a-48ec-9236-86afbe6258b1-proxy-tls\") pod \"isvc-canary-rollback-predictor-b4645bb5c-95nq7\" (UID: \"ddb77b27-527a-48ec-9236-86afbe6258b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"
Jul 28 17:40:37.020228 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.020096    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kcwz\" (UniqueName: \"kubernetes.io/projected/ddb77b27-527a-48ec-9236-86afbe6258b1-kube-api-access-6kcwz\") pod \"isvc-canary-rollback-predictor-b4645bb5c-95nq7\" (UID: \"ddb77b27-527a-48ec-9236-86afbe6258b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"
Jul 28 17:40:37.020228 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.020142    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddb77b27-527a-48ec-9236-86afbe6258b1-cabundle-cert\") pod \"isvc-canary-rollback-predictor-b4645bb5c-95nq7\" (UID: \"ddb77b27-527a-48ec-9236-86afbe6258b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"
Jul 28 17:40:37.020347 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.020216    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac024731-75c0-4bc1-8dcf-b09bdc17c930-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w\" (UID: \"ac024731-75c0-4bc1-8dcf-b09bdc17c930\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"
Jul 28 17:40:37.020347 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.020270    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdzmr\" (UniqueName: \"kubernetes.io/projected/ac024731-75c0-4bc1-8dcf-b09bdc17c930-kube-api-access-pdzmr\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w\" (UID: \"ac024731-75c0-4bc1-8dcf-b09bdc17c930\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"
Jul 28 17:40:37.020347 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.020316    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac024731-75c0-4bc1-8dcf-b09bdc17c930-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w\" (UID: \"ac024731-75c0-4bc1-8dcf-b09bdc17c930\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"
Jul 28 17:40:37.020458 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.020374    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddb77b27-527a-48ec-9236-86afbe6258b1-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-b4645bb5c-95nq7\" (UID: \"ddb77b27-527a-48ec-9236-86afbe6258b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"
Jul 28 17:40:37.020458 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.020407    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddb77b27-527a-48ec-9236-86afbe6258b1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-b4645bb5c-95nq7\" (UID: \"ddb77b27-527a-48ec-9236-86afbe6258b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"
Jul 28 17:40:37.020458 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.020434    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac024731-75c0-4bc1-8dcf-b09bdc17c930-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w\" (UID: \"ac024731-75c0-4bc1-8dcf-b09bdc17c930\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"
Jul 28 17:40:37.020568 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.020492    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac024731-75c0-4bc1-8dcf-b09bdc17c930-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w\" (UID: \"ac024731-75c0-4bc1-8dcf-b09bdc17c930\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"
Jul 28 17:40:37.091244 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.091203    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:40:37.121284 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.121260    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddb77b27-527a-48ec-9236-86afbe6258b1-cabundle-cert\") pod \"isvc-canary-rollback-predictor-b4645bb5c-95nq7\" (UID: \"ddb77b27-527a-48ec-9236-86afbe6258b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"
Jul 28 17:40:37.121414 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.121298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac024731-75c0-4bc1-8dcf-b09bdc17c930-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w\" (UID: \"ac024731-75c0-4bc1-8dcf-b09bdc17c930\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"
Jul 28 17:40:37.121414 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.121323    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdzmr\" (UniqueName: \"kubernetes.io/projected/ac024731-75c0-4bc1-8dcf-b09bdc17c930-kube-api-access-pdzmr\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w\" (UID: \"ac024731-75c0-4bc1-8dcf-b09bdc17c930\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"
Jul 28 17:40:37.121414 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.121339    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac024731-75c0-4bc1-8dcf-b09bdc17c930-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w\" (UID: \"ac024731-75c0-4bc1-8dcf-b09bdc17c930\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"
Jul 28 17:40:37.121414 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.121359    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddb77b27-527a-48ec-9236-86afbe6258b1-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-b4645bb5c-95nq7\" (UID: \"ddb77b27-527a-48ec-9236-86afbe6258b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"
Jul 28 17:40:37.121414 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.121377    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddb77b27-527a-48ec-9236-86afbe6258b1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-b4645bb5c-95nq7\" (UID: \"ddb77b27-527a-48ec-9236-86afbe6258b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"
Jul 28 17:40:37.121414 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.121404    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac024731-75c0-4bc1-8dcf-b09bdc17c930-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w\" (UID: \"ac024731-75c0-4bc1-8dcf-b09bdc17c930\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"
Jul 28 17:40:37.121712 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.121429    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac024731-75c0-4bc1-8dcf-b09bdc17c930-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w\" (UID: \"ac024731-75c0-4bc1-8dcf-b09bdc17c930\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"
Jul 28 17:40:37.121712 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.121454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddb77b27-527a-48ec-9236-86afbe6258b1-proxy-tls\") pod \"isvc-canary-rollback-predictor-b4645bb5c-95nq7\" (UID: \"ddb77b27-527a-48ec-9236-86afbe6258b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"
Jul 28 17:40:37.121853 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.121831    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddb77b27-527a-48ec-9236-86afbe6258b1-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-b4645bb5c-95nq7\" (UID: \"ddb77b27-527a-48ec-9236-86afbe6258b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"
Jul 28 17:40:37.122068 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.122050    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kcwz\" (UniqueName: \"kubernetes.io/projected/ddb77b27-527a-48ec-9236-86afbe6258b1-kube-api-access-6kcwz\") pod \"isvc-canary-rollback-predictor-b4645bb5c-95nq7\" (UID: \"ddb77b27-527a-48ec-9236-86afbe6258b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"
Jul 28 17:40:37.122182 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.122157    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac024731-75c0-4bc1-8dcf-b09bdc17c930-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w\" (UID: \"ac024731-75c0-4bc1-8dcf-b09bdc17c930\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"
Jul 28 17:40:37.122182 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.122085    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddb77b27-527a-48ec-9236-86afbe6258b1-cabundle-cert\") pod \"isvc-canary-rollback-predictor-b4645bb5c-95nq7\" (UID: \"ddb77b27-527a-48ec-9236-86afbe6258b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"
Jul 28 17:40:37.122350 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.122085    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddb77b27-527a-48ec-9236-86afbe6258b1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-b4645bb5c-95nq7\" (UID: \"ddb77b27-527a-48ec-9236-86afbe6258b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"
Jul 28 17:40:37.122350 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.122073    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac024731-75c0-4bc1-8dcf-b09bdc17c930-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w\" (UID: \"ac024731-75c0-4bc1-8dcf-b09bdc17c930\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"
Jul 28 17:40:37.122350 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:40:37.122215    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 28 17:40:37.122350 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:40:37.122321    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac024731-75c0-4bc1-8dcf-b09bdc17c930-proxy-tls podName:ac024731-75c0-4bc1-8dcf-b09bdc17c930 nodeName:}" failed. No retries permitted until 2026-07-28 17:40:37.622287471 +0000 UTC m=+1885.896903897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ac024731-75c0-4bc1-8dcf-b09bdc17c930-proxy-tls") pod "isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w" (UID: "ac024731-75c0-4bc1-8dcf-b09bdc17c930") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 28 17:40:37.122551 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.122438    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac024731-75c0-4bc1-8dcf-b09bdc17c930-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w\" (UID: \"ac024731-75c0-4bc1-8dcf-b09bdc17c930\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"
Jul 28 17:40:37.123927 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.123911    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddb77b27-527a-48ec-9236-86afbe6258b1-proxy-tls\") pod \"isvc-canary-rollback-predictor-b4645bb5c-95nq7\" (UID: \"ddb77b27-527a-48ec-9236-86afbe6258b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"
Jul 28 17:40:37.133336 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.133279    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kcwz\" (UniqueName: \"kubernetes.io/projected/ddb77b27-527a-48ec-9236-86afbe6258b1-kube-api-access-6kcwz\") pod \"isvc-canary-rollback-predictor-b4645bb5c-95nq7\" (UID: \"ddb77b27-527a-48ec-9236-86afbe6258b1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"
Jul 28 17:40:37.133424 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.133370    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdzmr\" (UniqueName: \"kubernetes.io/projected/ac024731-75c0-4bc1-8dcf-b09bdc17c930-kube-api-access-pdzmr\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w\" (UID: \"ac024731-75c0-4bc1-8dcf-b09bdc17c930\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"
Jul 28 17:40:37.263687 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.263656    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"
Jul 28 17:40:37.392597 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.392520    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"]
Jul 28 17:40:37.396143 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:40:37.396116    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb77b27_527a_48ec_9236_86afbe6258b1.slice/crio-16ee67de1af0f9053e86dad03bc724920bd55d6d4fdc8beec558539038b1dc9c WatchSource:0}: Error finding container 16ee67de1af0f9053e86dad03bc724920bd55d6d4fdc8beec558539038b1dc9c: Status 404 returned error can't find the container with id 16ee67de1af0f9053e86dad03bc724920bd55d6d4fdc8beec558539038b1dc9c
Jul 28 17:40:37.524318 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.524262    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:40:37.626913 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.626881    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac024731-75c0-4bc1-8dcf-b09bdc17c930-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w\" (UID: \"ac024731-75c0-4bc1-8dcf-b09bdc17c930\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"
Jul 28 17:40:37.629190 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.629163    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac024731-75c0-4bc1-8dcf-b09bdc17c930-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w\" (UID: \"ac024731-75c0-4bc1-8dcf-b09bdc17c930\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"
Jul 28 17:40:37.874071 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:37.874030    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"
Jul 28 17:40:38.003191 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:38.003144    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"]
Jul 28 17:40:38.003794 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:38.003766    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7" event={"ID":"ddb77b27-527a-48ec-9236-86afbe6258b1","Type":"ContainerStarted","Data":"6e8b6775dbdd722036fc56ca07fbe724d9039e8df315995747a77dbc82ba1b20"}
Jul 28 17:40:38.003865 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:38.003825    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7" event={"ID":"ddb77b27-527a-48ec-9236-86afbe6258b1","Type":"ContainerStarted","Data":"16ee67de1af0f9053e86dad03bc724920bd55d6d4fdc8beec558539038b1dc9c"}
Jul 28 17:40:38.006019 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:40:38.005999    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac024731_75c0_4bc1_8dcf_b09bdc17c930.slice/crio-bd97b3c7463f66a7555312abdfb9c8c19c12c60fb7d23bf733a2c81032faaf4a WatchSource:0}: Error finding container bd97b3c7463f66a7555312abdfb9c8c19c12c60fb7d23bf733a2c81032faaf4a: Status 404 returned error can't find the container with id bd97b3c7463f66a7555312abdfb9c8c19c12c60fb7d23bf733a2c81032faaf4a
Jul 28 17:40:38.290890 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:38.290852    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88b59350-8ec2-439c-91c5-f80ffc0b531a" path="/var/lib/kubelet/pods/88b59350-8ec2-439c-91c5-f80ffc0b531a/volumes"
Jul 28 17:40:39.008858 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:39.008820    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w" event={"ID":"ac024731-75c0-4bc1-8dcf-b09bdc17c930","Type":"ContainerStarted","Data":"4b1c3ec01891448c687a6dfb6084444f141def9ee7fc6a57db8744c9d4ff247d"}
Jul 28 17:40:39.009343 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:39.008886    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w" event={"ID":"ac024731-75c0-4bc1-8dcf-b09bdc17c930","Type":"ContainerStarted","Data":"bd97b3c7463f66a7555312abdfb9c8c19c12c60fb7d23bf733a2c81032faaf4a"}
Jul 28 17:40:42.026472 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:42.026385    2581 generic.go:356] "Generic (PLEG): container finished" podID="ddb77b27-527a-48ec-9236-86afbe6258b1" containerID="6e8b6775dbdd722036fc56ca07fbe724d9039e8df315995747a77dbc82ba1b20" exitCode=0
Jul 28 17:40:42.026906 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:42.026461    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7" event={"ID":"ddb77b27-527a-48ec-9236-86afbe6258b1","Type":"ContainerDied","Data":"6e8b6775dbdd722036fc56ca07fbe724d9039e8df315995747a77dbc82ba1b20"}
Jul 28 17:40:42.028062 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:42.028042    2581 generic.go:356] "Generic (PLEG): container finished" podID="ac024731-75c0-4bc1-8dcf-b09bdc17c930" containerID="4b1c3ec01891448c687a6dfb6084444f141def9ee7fc6a57db8744c9d4ff247d" exitCode=0
Jul 28 17:40:42.028159 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:42.028090    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w" event={"ID":"ac024731-75c0-4bc1-8dcf-b09bdc17c930","Type":"ContainerDied","Data":"4b1c3ec01891448c687a6dfb6084444f141def9ee7fc6a57db8744c9d4ff247d"}
Jul 28 17:40:43.033419 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:43.033381    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7" event={"ID":"ddb77b27-527a-48ec-9236-86afbe6258b1","Type":"ContainerStarted","Data":"37a6e31637941bba452e2ac8edb7f1cafc8010a1f63855c5217523cf8ec4b70d"}
Jul 28 17:40:43.033828 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:43.033455    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7" event={"ID":"ddb77b27-527a-48ec-9236-86afbe6258b1","Type":"ContainerStarted","Data":"42cd6cb079e6af8810494d0b835390530bc0b381c2416180d4e8dde6773cc33f"}
Jul 28 17:40:43.033828 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:43.033762    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"
Jul 28 17:40:43.033989 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:43.033963    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"
Jul 28 17:40:43.035418 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:43.035393    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7" podUID="ddb77b27-527a-48ec-9236-86afbe6258b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:40:43.035521 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:43.035458    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w" event={"ID":"ac024731-75c0-4bc1-8dcf-b09bdc17c930","Type":"ContainerStarted","Data":"ca3b0bffa1cc7704c3c8e77d1847b6a2afa7daa6988d23e184d72aea1af96720"}
Jul 28 17:40:43.035521 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:43.035506    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w" event={"ID":"ac024731-75c0-4bc1-8dcf-b09bdc17c930","Type":"ContainerStarted","Data":"2b37d799bfeb5f73ef01e483b5b19e8833e5b9c674c76bc0c914a916da3b74f5"}
Jul 28 17:40:43.035710 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:43.035693    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"
Jul 28 17:40:43.035766 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:43.035740    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"
Jul 28 17:40:43.036702 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:43.036677    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w" podUID="ac024731-75c0-4bc1-8dcf-b09bdc17c930" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:40:43.055165 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:43.055119    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7" podStartSLOduration=3.213138916 podStartE2EDuration="7.055104921s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.841966005s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:40:43.053700857 +0000 UTC m=+1891.328317293" watchObservedRunningTime="2026-07-28 17:40:43.055104921 +0000 UTC m=+1891.329721354"
Jul 28 17:40:44.038726 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:44.038683    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7" podUID="ddb77b27-527a-48ec-9236-86afbe6258b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:40:44.039153 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:44.038811    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w" podUID="ac024731-75c0-4bc1-8dcf-b09bdc17c930" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:40:45.042182 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:45.042142    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7" podUID="ddb77b27-527a-48ec-9236-86afbe6258b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:40:49.043416 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:49.043388    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"
Jul 28 17:40:49.043991 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:49.043960    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w" podUID="ac024731-75c0-4bc1-8dcf-b09bdc17c930" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:40:49.065079 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:49.065023    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w" podStartSLOduration=9.474963304 podStartE2EDuration="13.065008302s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.590044998s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:40:43.073528179 +0000 UTC m=+1891.348144635" watchObservedRunningTime="2026-07-28 17:40:49.065008302 +0000 UTC m=+1897.339624733"
Jul 28 17:40:50.047078 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:50.047046    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"
Jul 28 17:40:50.047693 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:50.047665    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7" podUID="ddb77b27-527a-48ec-9236-86afbe6258b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:40:59.044269 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:40:59.044227    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w" podUID="ac024731-75c0-4bc1-8dcf-b09bdc17c930" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:41:00.047876 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:41:00.047832    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7" podUID="ddb77b27-527a-48ec-9236-86afbe6258b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:41:09.044826 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:41:09.044786    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w" podUID="ac024731-75c0-4bc1-8dcf-b09bdc17c930" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:41:10.047975 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:41:10.047918    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7" podUID="ddb77b27-527a-48ec-9236-86afbe6258b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:41:19.044494 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:41:19.044454    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w" podUID="ac024731-75c0-4bc1-8dcf-b09bdc17c930" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:41:20.048365 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:41:20.048320    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7" podUID="ddb77b27-527a-48ec-9236-86afbe6258b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:41:29.043826 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:41:29.043789    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w" podUID="ac024731-75c0-4bc1-8dcf-b09bdc17c930" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:41:30.047979 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:41:30.047918    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7" podUID="ddb77b27-527a-48ec-9236-86afbe6258b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:41:39.044531 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:41:39.044494    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w" podUID="ac024731-75c0-4bc1-8dcf-b09bdc17c930" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:41:40.048473 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:41:40.048433    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7" podUID="ddb77b27-527a-48ec-9236-86afbe6258b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:41:45.149236 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:41:45.149197    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:41:49.044537 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:41:49.044494    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w" podUID="ac024731-75c0-4bc1-8dcf-b09bdc17c930" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:41:50.047574 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:41:50.047536    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7" podUID="ddb77b27-527a-48ec-9236-86afbe6258b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:41:59.045101 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:41:59.045066    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"
Jul 28 17:42:00.048657 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:00.048627    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"
Jul 28 17:42:07.010986 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:07.010888    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"]
Jul 28 17:42:07.011529 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:07.011220    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w" podUID="ac024731-75c0-4bc1-8dcf-b09bdc17c930" containerName="kserve-container" containerID="cri-o://2b37d799bfeb5f73ef01e483b5b19e8833e5b9c674c76bc0c914a916da3b74f5" gracePeriod=30
Jul 28 17:42:07.011529 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:07.011298    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w" podUID="ac024731-75c0-4bc1-8dcf-b09bdc17c930" containerName="kube-rbac-proxy" containerID="cri-o://ca3b0bffa1cc7704c3c8e77d1847b6a2afa7daa6988d23e184d72aea1af96720" gracePeriod=30
Jul 28 17:42:07.035277 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:07.035249    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:42:07.334746 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:07.334658    2581 generic.go:356] "Generic (PLEG): container finished" podID="ac024731-75c0-4bc1-8dcf-b09bdc17c930" containerID="ca3b0bffa1cc7704c3c8e77d1847b6a2afa7daa6988d23e184d72aea1af96720" exitCode=2
Jul 28 17:42:07.334746 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:07.334729    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w" event={"ID":"ac024731-75c0-4bc1-8dcf-b09bdc17c930","Type":"ContainerDied","Data":"ca3b0bffa1cc7704c3c8e77d1847b6a2afa7daa6988d23e184d72aea1af96720"}
Jul 28 17:42:09.039431 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:09.039391    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w" podUID="ac024731-75c0-4bc1-8dcf-b09bdc17c930" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 28 17:42:09.044795 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:09.044769    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w" podUID="ac024731-75c0-4bc1-8dcf-b09bdc17c930" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:42:11.349738 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:11.349712    2581 generic.go:356] "Generic (PLEG): container finished" podID="ac024731-75c0-4bc1-8dcf-b09bdc17c930" containerID="2b37d799bfeb5f73ef01e483b5b19e8833e5b9c674c76bc0c914a916da3b74f5" exitCode=0
Jul 28 17:42:11.350097 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:11.349776    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w" event={"ID":"ac024731-75c0-4bc1-8dcf-b09bdc17c930","Type":"ContainerDied","Data":"2b37d799bfeb5f73ef01e483b5b19e8833e5b9c674c76bc0c914a916da3b74f5"}
Jul 28 17:42:11.350097 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:11.349852    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w" event={"ID":"ac024731-75c0-4bc1-8dcf-b09bdc17c930","Type":"ContainerDied","Data":"bd97b3c7463f66a7555312abdfb9c8c19c12c60fb7d23bf733a2c81032faaf4a"}
Jul 28 17:42:11.350097 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:11.349888    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd97b3c7463f66a7555312abdfb9c8c19c12c60fb7d23bf733a2c81032faaf4a"
Jul 28 17:42:11.358820 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:11.358802    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"
Jul 28 17:42:11.416602 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:11.416517    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac024731-75c0-4bc1-8dcf-b09bdc17c930-cabundle-cert\") pod \"ac024731-75c0-4bc1-8dcf-b09bdc17c930\" (UID: \"ac024731-75c0-4bc1-8dcf-b09bdc17c930\") "
Jul 28 17:42:11.416602 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:11.416560    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac024731-75c0-4bc1-8dcf-b09bdc17c930-proxy-tls\") pod \"ac024731-75c0-4bc1-8dcf-b09bdc17c930\" (UID: \"ac024731-75c0-4bc1-8dcf-b09bdc17c930\") "
Jul 28 17:42:11.416602 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:11.416579    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdzmr\" (UniqueName: \"kubernetes.io/projected/ac024731-75c0-4bc1-8dcf-b09bdc17c930-kube-api-access-pdzmr\") pod \"ac024731-75c0-4bc1-8dcf-b09bdc17c930\" (UID: \"ac024731-75c0-4bc1-8dcf-b09bdc17c930\") "
Jul 28 17:42:11.416875 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:11.416620    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac024731-75c0-4bc1-8dcf-b09bdc17c930-kserve-provision-location\") pod \"ac024731-75c0-4bc1-8dcf-b09bdc17c930\" (UID: \"ac024731-75c0-4bc1-8dcf-b09bdc17c930\") "
Jul 28 17:42:11.416875 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:11.416658    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac024731-75c0-4bc1-8dcf-b09bdc17c930-kube-rbac-proxy-sar-config\") pod \"ac024731-75c0-4bc1-8dcf-b09bdc17c930\" (UID: \"ac024731-75c0-4bc1-8dcf-b09bdc17c930\") "
Jul 28 17:42:11.417004 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:11.416971    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac024731-75c0-4bc1-8dcf-b09bdc17c930-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ac024731-75c0-4bc1-8dcf-b09bdc17c930" (UID: "ac024731-75c0-4bc1-8dcf-b09bdc17c930"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:42:11.417049 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:11.417017    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac024731-75c0-4bc1-8dcf-b09bdc17c930-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ac024731-75c0-4bc1-8dcf-b09bdc17c930" (UID: "ac024731-75c0-4bc1-8dcf-b09bdc17c930"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:42:11.417085 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:11.417039    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac024731-75c0-4bc1-8dcf-b09bdc17c930-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ac024731-75c0-4bc1-8dcf-b09bdc17c930" (UID: "ac024731-75c0-4bc1-8dcf-b09bdc17c930"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:42:11.418644 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:11.418622    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac024731-75c0-4bc1-8dcf-b09bdc17c930-kube-api-access-pdzmr" (OuterVolumeSpecName: "kube-api-access-pdzmr") pod "ac024731-75c0-4bc1-8dcf-b09bdc17c930" (UID: "ac024731-75c0-4bc1-8dcf-b09bdc17c930"). InnerVolumeSpecName "kube-api-access-pdzmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:42:11.418716 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:11.418638    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac024731-75c0-4bc1-8dcf-b09bdc17c930-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ac024731-75c0-4bc1-8dcf-b09bdc17c930" (UID: "ac024731-75c0-4bc1-8dcf-b09bdc17c930"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:42:11.517203 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:11.517167    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac024731-75c0-4bc1-8dcf-b09bdc17c930-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:42:11.517203 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:11.517195    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac024731-75c0-4bc1-8dcf-b09bdc17c930-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:42:11.517203 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:11.517207    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac024731-75c0-4bc1-8dcf-b09bdc17c930-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:42:11.517436 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:11.517215    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdzmr\" (UniqueName: \"kubernetes.io/projected/ac024731-75c0-4bc1-8dcf-b09bdc17c930-kube-api-access-pdzmr\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:42:11.517436 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:11.517224    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac024731-75c0-4bc1-8dcf-b09bdc17c930-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:42:12.353416 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:12.353388    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"
Jul 28 17:42:12.372582 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:12.372555    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"]
Jul 28 17:42:12.376100 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:12.376073    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-648cbdfbd8-jv69w"]
Jul 28 17:42:14.289724 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:42:14.289688    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac024731-75c0-4bc1-8dcf-b09bdc17c930" path="/var/lib/kubelet/pods/ac024731-75c0-4bc1-8dcf-b09bdc17c930/volumes"
Jul 28 17:43:37.315397 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:37.315313    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"]
Jul 28 17:43:37.315977 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:37.315619    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7" podUID="ddb77b27-527a-48ec-9236-86afbe6258b1" containerName="kserve-container" containerID="cri-o://42cd6cb079e6af8810494d0b835390530bc0b381c2416180d4e8dde6773cc33f" gracePeriod=30
Jul 28 17:43:37.315977 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:37.315640    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7" podUID="ddb77b27-527a-48ec-9236-86afbe6258b1" containerName="kube-rbac-proxy" containerID="cri-o://37a6e31637941bba452e2ac8edb7f1cafc8010a1f63855c5217523cf8ec4b70d" gracePeriod=30
Jul 28 17:43:37.361484 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:37.361457    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:37.643275 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:37.643192    2581 generic.go:356] "Generic (PLEG): container finished" podID="ddb77b27-527a-48ec-9236-86afbe6258b1" containerID="37a6e31637941bba452e2ac8edb7f1cafc8010a1f63855c5217523cf8ec4b70d" exitCode=2
Jul 28 17:43:37.643421 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:37.643273    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7" event={"ID":"ddb77b27-527a-48ec-9236-86afbe6258b1","Type":"ContainerDied","Data":"37a6e31637941bba452e2ac8edb7f1cafc8010a1f63855c5217523cf8ec4b70d"}
Jul 28 17:43:40.043442 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:40.043409    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7" podUID="ddb77b27-527a-48ec-9236-86afbe6258b1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 28 17:43:40.047736 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:40.047709    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7" podUID="ddb77b27-527a-48ec-9236-86afbe6258b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:43:41.652232 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.652205    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"
Jul 28 17:43:41.658400 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.658376    2581 generic.go:356] "Generic (PLEG): container finished" podID="ddb77b27-527a-48ec-9236-86afbe6258b1" containerID="42cd6cb079e6af8810494d0b835390530bc0b381c2416180d4e8dde6773cc33f" exitCode=0
Jul 28 17:43:41.658490 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.658421    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"
Jul 28 17:43:41.658490 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.658453    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7" event={"ID":"ddb77b27-527a-48ec-9236-86afbe6258b1","Type":"ContainerDied","Data":"42cd6cb079e6af8810494d0b835390530bc0b381c2416180d4e8dde6773cc33f"}
Jul 28 17:43:41.658559 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.658514    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7" event={"ID":"ddb77b27-527a-48ec-9236-86afbe6258b1","Type":"ContainerDied","Data":"16ee67de1af0f9053e86dad03bc724920bd55d6d4fdc8beec558539038b1dc9c"}
Jul 28 17:43:41.658559 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.658552    2581 scope.go:117] "RemoveContainer" containerID="37a6e31637941bba452e2ac8edb7f1cafc8010a1f63855c5217523cf8ec4b70d"
Jul 28 17:43:41.665758 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.665740    2581 scope.go:117] "RemoveContainer" containerID="42cd6cb079e6af8810494d0b835390530bc0b381c2416180d4e8dde6773cc33f"
Jul 28 17:43:41.672486 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.672470    2581 scope.go:117] "RemoveContainer" containerID="6e8b6775dbdd722036fc56ca07fbe724d9039e8df315995747a77dbc82ba1b20"
Jul 28 17:43:41.680059 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.680035    2581 scope.go:117] "RemoveContainer" containerID="37a6e31637941bba452e2ac8edb7f1cafc8010a1f63855c5217523cf8ec4b70d"
Jul 28 17:43:41.680360 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:43:41.680339    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37a6e31637941bba452e2ac8edb7f1cafc8010a1f63855c5217523cf8ec4b70d\": container with ID starting with 37a6e31637941bba452e2ac8edb7f1cafc8010a1f63855c5217523cf8ec4b70d not found: ID does not exist" containerID="37a6e31637941bba452e2ac8edb7f1cafc8010a1f63855c5217523cf8ec4b70d"
Jul 28 17:43:41.680412 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.680367    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37a6e31637941bba452e2ac8edb7f1cafc8010a1f63855c5217523cf8ec4b70d"} err="failed to get container status \"37a6e31637941bba452e2ac8edb7f1cafc8010a1f63855c5217523cf8ec4b70d\": rpc error: code = NotFound desc = could not find container \"37a6e31637941bba452e2ac8edb7f1cafc8010a1f63855c5217523cf8ec4b70d\": container with ID starting with 37a6e31637941bba452e2ac8edb7f1cafc8010a1f63855c5217523cf8ec4b70d not found: ID does not exist"
Jul 28 17:43:41.680412 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.680384    2581 scope.go:117] "RemoveContainer" containerID="42cd6cb079e6af8810494d0b835390530bc0b381c2416180d4e8dde6773cc33f"
Jul 28 17:43:41.680624 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:43:41.680603    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42cd6cb079e6af8810494d0b835390530bc0b381c2416180d4e8dde6773cc33f\": container with ID starting with 42cd6cb079e6af8810494d0b835390530bc0b381c2416180d4e8dde6773cc33f not found: ID does not exist" containerID="42cd6cb079e6af8810494d0b835390530bc0b381c2416180d4e8dde6773cc33f"
Jul 28 17:43:41.680699 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.680635    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42cd6cb079e6af8810494d0b835390530bc0b381c2416180d4e8dde6773cc33f"} err="failed to get container status \"42cd6cb079e6af8810494d0b835390530bc0b381c2416180d4e8dde6773cc33f\": rpc error: code = NotFound desc = could not find container \"42cd6cb079e6af8810494d0b835390530bc0b381c2416180d4e8dde6773cc33f\": container with ID starting with 42cd6cb079e6af8810494d0b835390530bc0b381c2416180d4e8dde6773cc33f not found: ID does not exist"
Jul 28 17:43:41.680699 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.680658    2581 scope.go:117] "RemoveContainer" containerID="6e8b6775dbdd722036fc56ca07fbe724d9039e8df315995747a77dbc82ba1b20"
Jul 28 17:43:41.680992 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:43:41.680972    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e8b6775dbdd722036fc56ca07fbe724d9039e8df315995747a77dbc82ba1b20\": container with ID starting with 6e8b6775dbdd722036fc56ca07fbe724d9039e8df315995747a77dbc82ba1b20 not found: ID does not exist" containerID="6e8b6775dbdd722036fc56ca07fbe724d9039e8df315995747a77dbc82ba1b20"
Jul 28 17:43:41.681074 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.680995    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e8b6775dbdd722036fc56ca07fbe724d9039e8df315995747a77dbc82ba1b20"} err="failed to get container status \"6e8b6775dbdd722036fc56ca07fbe724d9039e8df315995747a77dbc82ba1b20\": rpc error: code = NotFound desc = could not find container \"6e8b6775dbdd722036fc56ca07fbe724d9039e8df315995747a77dbc82ba1b20\": container with ID starting with 6e8b6775dbdd722036fc56ca07fbe724d9039e8df315995747a77dbc82ba1b20 not found: ID does not exist"
Jul 28 17:43:41.734897 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.734835    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddb77b27-527a-48ec-9236-86afbe6258b1-cabundle-cert\") pod \"ddb77b27-527a-48ec-9236-86afbe6258b1\" (UID: \"ddb77b27-527a-48ec-9236-86afbe6258b1\") "
Jul 28 17:43:41.734897 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.734875    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kcwz\" (UniqueName: \"kubernetes.io/projected/ddb77b27-527a-48ec-9236-86afbe6258b1-kube-api-access-6kcwz\") pod \"ddb77b27-527a-48ec-9236-86afbe6258b1\" (UID: \"ddb77b27-527a-48ec-9236-86afbe6258b1\") "
Jul 28 17:43:41.735091 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.734917    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddb77b27-527a-48ec-9236-86afbe6258b1-kserve-provision-location\") pod \"ddb77b27-527a-48ec-9236-86afbe6258b1\" (UID: \"ddb77b27-527a-48ec-9236-86afbe6258b1\") "
Jul 28 17:43:41.735091 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.734977    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddb77b27-527a-48ec-9236-86afbe6258b1-kube-rbac-proxy-sar-config\") pod \"ddb77b27-527a-48ec-9236-86afbe6258b1\" (UID: \"ddb77b27-527a-48ec-9236-86afbe6258b1\") "
Jul 28 17:43:41.735208 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.735164    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddb77b27-527a-48ec-9236-86afbe6258b1-proxy-tls\") pod \"ddb77b27-527a-48ec-9236-86afbe6258b1\" (UID: \"ddb77b27-527a-48ec-9236-86afbe6258b1\") "
Jul 28 17:43:41.735272 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.735234    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddb77b27-527a-48ec-9236-86afbe6258b1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ddb77b27-527a-48ec-9236-86afbe6258b1" (UID: "ddb77b27-527a-48ec-9236-86afbe6258b1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:43:41.735329 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.735273    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb77b27-527a-48ec-9236-86afbe6258b1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ddb77b27-527a-48ec-9236-86afbe6258b1" (UID: "ddb77b27-527a-48ec-9236-86afbe6258b1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:43:41.735329 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.735300    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddb77b27-527a-48ec-9236-86afbe6258b1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ddb77b27-527a-48ec-9236-86afbe6258b1" (UID: "ddb77b27-527a-48ec-9236-86afbe6258b1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:43:41.735417 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.735370    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddb77b27-527a-48ec-9236-86afbe6258b1-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:43:41.735417 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.735383    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddb77b27-527a-48ec-9236-86afbe6258b1-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:43:41.735417 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.735393    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddb77b27-527a-48ec-9236-86afbe6258b1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:43:41.737047 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.737025    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddb77b27-527a-48ec-9236-86afbe6258b1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ddb77b27-527a-48ec-9236-86afbe6258b1" (UID: "ddb77b27-527a-48ec-9236-86afbe6258b1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:43:41.737136 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.737112    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb77b27-527a-48ec-9236-86afbe6258b1-kube-api-access-6kcwz" (OuterVolumeSpecName: "kube-api-access-6kcwz") pod "ddb77b27-527a-48ec-9236-86afbe6258b1" (UID: "ddb77b27-527a-48ec-9236-86afbe6258b1"). InnerVolumeSpecName "kube-api-access-6kcwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:43:41.835927 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.835898    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kcwz\" (UniqueName: \"kubernetes.io/projected/ddb77b27-527a-48ec-9236-86afbe6258b1-kube-api-access-6kcwz\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:43:41.835927 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.835924    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddb77b27-527a-48ec-9236-86afbe6258b1-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:43:41.983240 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.983210    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"]
Jul 28 17:43:41.987175 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:41.987114    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-95nq7"]
Jul 28 17:43:42.291308 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.291215    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddb77b27-527a-48ec-9236-86afbe6258b1" path="/var/lib/kubelet/pods/ddb77b27-527a-48ec-9236-86afbe6258b1/volumes"
Jul 28 17:43:42.327392 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.327345    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:42.408399 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.408367    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"]
Jul 28 17:43:42.408703 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.408687    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac024731-75c0-4bc1-8dcf-b09bdc17c930" containerName="kserve-container"
Jul 28 17:43:42.408787 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.408705    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac024731-75c0-4bc1-8dcf-b09bdc17c930" containerName="kserve-container"
Jul 28 17:43:42.408787 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.408730    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddb77b27-527a-48ec-9236-86afbe6258b1" containerName="storage-initializer"
Jul 28 17:43:42.408787 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.408738    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddb77b27-527a-48ec-9236-86afbe6258b1" containerName="storage-initializer"
Jul 28 17:43:42.408787 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.408748    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac024731-75c0-4bc1-8dcf-b09bdc17c930" containerName="kube-rbac-proxy"
Jul 28 17:43:42.408787 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.408756    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac024731-75c0-4bc1-8dcf-b09bdc17c930" containerName="kube-rbac-proxy"
Jul 28 17:43:42.408787 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.408766    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddb77b27-527a-48ec-9236-86afbe6258b1" containerName="kube-rbac-proxy"
Jul 28 17:43:42.408787 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.408775    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddb77b27-527a-48ec-9236-86afbe6258b1" containerName="kube-rbac-proxy"
Jul 28 17:43:42.408787 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.408788    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddb77b27-527a-48ec-9236-86afbe6258b1" containerName="kserve-container"
Jul 28 17:43:42.409178 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.408796    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddb77b27-527a-48ec-9236-86afbe6258b1" containerName="kserve-container"
Jul 28 17:43:42.409178 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.408810    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac024731-75c0-4bc1-8dcf-b09bdc17c930" containerName="storage-initializer"
Jul 28 17:43:42.409178 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.408817    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac024731-75c0-4bc1-8dcf-b09bdc17c930" containerName="storage-initializer"
Jul 28 17:43:42.409178 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.408873    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddb77b27-527a-48ec-9236-86afbe6258b1" containerName="kube-rbac-proxy"
Jul 28 17:43:42.409178 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.408885    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddb77b27-527a-48ec-9236-86afbe6258b1" containerName="kserve-container"
Jul 28 17:43:42.409178 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.408893    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac024731-75c0-4bc1-8dcf-b09bdc17c930" containerName="kube-rbac-proxy"
Jul 28 17:43:42.409178 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.408901    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac024731-75c0-4bc1-8dcf-b09bdc17c930" containerName="kserve-container"
Jul 28 17:43:42.409841 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.409815    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:42.413320 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.413300    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"
Jul 28 17:43:42.416365 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.416343    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Jul 28 17:43:42.416483 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.416351    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 17:43:42.416483 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.416352    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:43:42.416483 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.416352    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 28 17:43:42.416685 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.416672    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:43:42.417508 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.417471    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:43:42.417619 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.417579    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:43:42.419677 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.419648    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"]
Jul 28 17:43:42.423246 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.423228    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"
Jul 28 17:43:42.425768 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.425744    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"]
Jul 28 17:43:42.425985 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.425969    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Jul 28 17:43:42.435228 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.435192    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"]
Jul 28 17:43:42.438846 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.438820    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h9md\" (UniqueName: \"kubernetes.io/projected/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-kube-api-access-2h9md\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb\" (UID: \"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"
Jul 28 17:43:42.439036 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.438847    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb\" (UID: \"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"
Jul 28 17:43:42.439036 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.438866    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5138a30-bf88-4515-a33f-6d955d4507b8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5bf5667989-44hnf\" (UID: \"f5138a30-bf88-4515-a33f-6d955d4507b8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"
Jul 28 17:43:42.439036 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.438896    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5138a30-bf88-4515-a33f-6d955d4507b8-cabundle-cert\") pod \"isvc-canary-stop-predictor-5bf5667989-44hnf\" (UID: \"f5138a30-bf88-4515-a33f-6d955d4507b8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"
Jul 28 17:43:42.439036 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.438922    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnkmd\" (UniqueName: \"kubernetes.io/projected/f5138a30-bf88-4515-a33f-6d955d4507b8-kube-api-access-dnkmd\") pod \"isvc-canary-stop-predictor-5bf5667989-44hnf\" (UID: \"f5138a30-bf88-4515-a33f-6d955d4507b8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"
Jul 28 17:43:42.439036 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.438958    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5138a30-bf88-4515-a33f-6d955d4507b8-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5bf5667989-44hnf\" (UID: \"f5138a30-bf88-4515-a33f-6d955d4507b8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"
Jul 28 17:43:42.439036 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.439000    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb\" (UID: \"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"
Jul 28 17:43:42.439036 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.439021    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb\" (UID: \"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"
Jul 28 17:43:42.439442 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.439070    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb\" (UID: \"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"
Jul 28 17:43:42.439442 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.439085    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5138a30-bf88-4515-a33f-6d955d4507b8-proxy-tls\") pod \"isvc-canary-stop-predictor-5bf5667989-44hnf\" (UID: \"f5138a30-bf88-4515-a33f-6d955d4507b8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"
Jul 28 17:43:42.469888 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.469863    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:42.539550 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.539515    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb\" (UID: \"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"
Jul 28 17:43:42.539737 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.539559    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb\" (UID: \"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"
Jul 28 17:43:42.539737 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.539606    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb\" (UID: \"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"
Jul 28 17:43:42.539737 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.539629    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5138a30-bf88-4515-a33f-6d955d4507b8-proxy-tls\") pod \"isvc-canary-stop-predictor-5bf5667989-44hnf\" (UID: \"f5138a30-bf88-4515-a33f-6d955d4507b8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"
Jul 28 17:43:42.539737 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.539660    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h9md\" (UniqueName: \"kubernetes.io/projected/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-kube-api-access-2h9md\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb\" (UID: \"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"
Jul 28 17:43:42.539737 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.539685    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb\" (UID: \"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"
Jul 28 17:43:42.539737 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.539707    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5138a30-bf88-4515-a33f-6d955d4507b8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5bf5667989-44hnf\" (UID: \"f5138a30-bf88-4515-a33f-6d955d4507b8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"
Jul 28 17:43:42.539737 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.539733    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5138a30-bf88-4515-a33f-6d955d4507b8-cabundle-cert\") pod \"isvc-canary-stop-predictor-5bf5667989-44hnf\" (UID: \"f5138a30-bf88-4515-a33f-6d955d4507b8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"
Jul 28 17:43:42.540183 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.539771    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnkmd\" (UniqueName: \"kubernetes.io/projected/f5138a30-bf88-4515-a33f-6d955d4507b8-kube-api-access-dnkmd\") pod \"isvc-canary-stop-predictor-5bf5667989-44hnf\" (UID: \"f5138a30-bf88-4515-a33f-6d955d4507b8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"
Jul 28 17:43:42.540183 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.539803    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5138a30-bf88-4515-a33f-6d955d4507b8-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5bf5667989-44hnf\" (UID: \"f5138a30-bf88-4515-a33f-6d955d4507b8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"
Jul 28 17:43:42.540183 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.539982    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb\" (UID: \"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"
Jul 28 17:43:42.540352 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.540242    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5138a30-bf88-4515-a33f-6d955d4507b8-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5bf5667989-44hnf\" (UID: \"f5138a30-bf88-4515-a33f-6d955d4507b8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"
Jul 28 17:43:42.540416 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.540368    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb\" (UID: \"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"
Jul 28 17:43:42.540480 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.540461    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb\" (UID: \"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"
Jul 28 17:43:42.540799 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.540776    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5138a30-bf88-4515-a33f-6d955d4507b8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5bf5667989-44hnf\" (UID: \"f5138a30-bf88-4515-a33f-6d955d4507b8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"
Jul 28 17:43:42.540962 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.540923    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5138a30-bf88-4515-a33f-6d955d4507b8-cabundle-cert\") pod \"isvc-canary-stop-predictor-5bf5667989-44hnf\" (UID: \"f5138a30-bf88-4515-a33f-6d955d4507b8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"
Jul 28 17:43:42.542080 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.542031    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5138a30-bf88-4515-a33f-6d955d4507b8-proxy-tls\") pod \"isvc-canary-stop-predictor-5bf5667989-44hnf\" (UID: \"f5138a30-bf88-4515-a33f-6d955d4507b8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"
Jul 28 17:43:42.542270 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.542252    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb\" (UID: \"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"
Jul 28 17:43:42.543007 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.542987    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:42.547585 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.547553    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h9md\" (UniqueName: \"kubernetes.io/projected/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-kube-api-access-2h9md\") pod \"isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb\" (UID: \"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"
Jul 28 17:43:42.547832 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.547802    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnkmd\" (UniqueName: \"kubernetes.io/projected/f5138a30-bf88-4515-a33f-6d955d4507b8-kube-api-access-dnkmd\") pod \"isvc-canary-stop-predictor-5bf5667989-44hnf\" (UID: \"f5138a30-bf88-4515-a33f-6d955d4507b8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"
Jul 28 17:43:42.725000 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.724972    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"
Jul 28 17:43:42.735131 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.734980    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"
Jul 28 17:43:42.859300 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.859268    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"]
Jul 28 17:43:42.862248 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:43:42.862222    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5138a30_bf88_4515_a33f_6d955d4507b8.slice/crio-4830e5a211e1a20c4937eb181a9a22998c73c3e58747f6b7b011c6bfff30f28e WatchSource:0}: Error finding container 4830e5a211e1a20c4937eb181a9a22998c73c3e58747f6b7b011c6bfff30f28e: Status 404 returned error can't find the container with id 4830e5a211e1a20c4937eb181a9a22998c73c3e58747f6b7b011c6bfff30f28e
Jul 28 17:43:42.878432 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:42.878410    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"]
Jul 28 17:43:42.880699 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:43:42.880671    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0eeff6d_1e9a_4d51_a705_b6a059ce6acd.slice/crio-abe4b2bc1f1b43b6f70f08ebc2578e99603973549d4f772fbcf13eea92ea3602 WatchSource:0}: Error finding container abe4b2bc1f1b43b6f70f08ebc2578e99603973549d4f772fbcf13eea92ea3602: Status 404 returned error can't find the container with id abe4b2bc1f1b43b6f70f08ebc2578e99603973549d4f772fbcf13eea92ea3602
Jul 28 17:43:43.666730 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:43.666690    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf" event={"ID":"f5138a30-bf88-4515-a33f-6d955d4507b8","Type":"ContainerStarted","Data":"37646a2dba678a97e09c847d39e1f849c51f7f7f654c8cbaa826a2fc06a1858b"}
Jul 28 17:43:43.666921 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:43.666746    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf" event={"ID":"f5138a30-bf88-4515-a33f-6d955d4507b8","Type":"ContainerStarted","Data":"4830e5a211e1a20c4937eb181a9a22998c73c3e58747f6b7b011c6bfff30f28e"}
Jul 28 17:43:43.668097 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:43.668073    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb" event={"ID":"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd","Type":"ContainerStarted","Data":"42bf868e64d50c897b328e18d97ecfa66c90c463e57bad181f747a4569743f91"}
Jul 28 17:43:43.668183 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:43.668122    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb" event={"ID":"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd","Type":"ContainerStarted","Data":"abe4b2bc1f1b43b6f70f08ebc2578e99603973549d4f772fbcf13eea92ea3602"}
Jul 28 17:43:46.679618 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:46.679585    2581 generic.go:356] "Generic (PLEG): container finished" podID="b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" containerID="42bf868e64d50c897b328e18d97ecfa66c90c463e57bad181f747a4569743f91" exitCode=0
Jul 28 17:43:46.680088 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:46.679642    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb" event={"ID":"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd","Type":"ContainerDied","Data":"42bf868e64d50c897b328e18d97ecfa66c90c463e57bad181f747a4569743f91"}
Jul 28 17:43:47.688004 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:47.687966    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb" event={"ID":"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd","Type":"ContainerStarted","Data":"5a2649206414963594d67723a2a7f847e0d7dc952f43ecef71c4554f2548f92d"}
Jul 28 17:43:47.688469 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:47.688043    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb" event={"ID":"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd","Type":"ContainerStarted","Data":"4a66addba5adcb920b69d9412a50a2c2a3b84dcc9069045b4fb2f8d39ca67164"}
Jul 28 17:43:47.688469 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:47.688229    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"
Jul 28 17:43:47.689344 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:47.689323    2581 generic.go:356] "Generic (PLEG): container finished" podID="f5138a30-bf88-4515-a33f-6d955d4507b8" containerID="37646a2dba678a97e09c847d39e1f849c51f7f7f654c8cbaa826a2fc06a1858b" exitCode=0
Jul 28 17:43:47.689442 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:47.689352    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf" event={"ID":"f5138a30-bf88-4515-a33f-6d955d4507b8","Type":"ContainerDied","Data":"37646a2dba678a97e09c847d39e1f849c51f7f7f654c8cbaa826a2fc06a1858b"}
Jul 28 17:43:47.711524 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:47.711484    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb" podStartSLOduration=2.011942149 podStartE2EDuration="5.711469037s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.699526888s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:47.709826012 +0000 UTC m=+2075.984442446" watchObservedRunningTime="2026-07-28 17:43:47.711469037 +0000 UTC m=+2075.986085469"
Jul 28 17:43:48.695643 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:48.695605    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf" event={"ID":"f5138a30-bf88-4515-a33f-6d955d4507b8","Type":"ContainerStarted","Data":"9a27f656f0c2af8adf44e4a76a2b5c0d799c50bf37684f17643c2000a346e2a4"}
Jul 28 17:43:48.696063 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:48.695835    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"
Jul 28 17:43:48.696063 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:48.695893    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf" event={"ID":"f5138a30-bf88-4515-a33f-6d955d4507b8","Type":"ContainerStarted","Data":"f944d1a7b416baac43fb538aac2e35ccbaf132f34c106893f4629b8ab73894b3"}
Jul 28 17:43:48.696153 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:48.696074    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"
Jul 28 17:43:48.697093 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:48.697071    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb" podUID="b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 17:43:48.718223 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:48.718178    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf" podStartSLOduration=2.822130881 podStartE2EDuration="6.718164433s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.896033552s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:48.715873264 +0000 UTC m=+2076.990489696" watchObservedRunningTime="2026-07-28 17:43:48.718164433 +0000 UTC m=+2076.992780920"
Jul 28 17:43:49.532804 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:49.532772    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:49.599550 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:49.599513    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:43:49.699003 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:49.698974    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"
Jul 28 17:43:49.699003 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:49.698991    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb" podUID="b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 17:43:49.700173 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:49.700152    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf" podUID="f5138a30-bf88-4515-a33f-6d955d4507b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 17:43:50.702678 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:50.702634    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf" podUID="f5138a30-bf88-4515-a33f-6d955d4507b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 17:43:54.703767 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:54.703735    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"
Jul 28 17:43:54.704266 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:54.704233    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb" podUID="b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 17:43:55.707778 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:55.707735    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"
Jul 28 17:43:55.708386 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:43:55.708355    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf" podUID="f5138a30-bf88-4515-a33f-6d955d4507b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 17:44:04.704918 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:44:04.704875    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb" podUID="b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 17:44:05.709124 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:44:05.709079    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf" podUID="f5138a30-bf88-4515-a33f-6d955d4507b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 17:44:12.297502 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:44:12.297473    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 17:44:12.299898 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:44:12.299874    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 17:44:14.704857 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:44:14.704819    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb" podUID="b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 17:44:15.709035 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:44:15.708997    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf" podUID="f5138a30-bf88-4515-a33f-6d955d4507b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 17:44:24.705151 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:44:24.705111    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb" podUID="b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 17:44:25.709001 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:44:25.708961    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf" podUID="f5138a30-bf88-4515-a33f-6d955d4507b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 17:44:34.704937 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:44:34.704894    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb" podUID="b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 17:44:35.708566 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:44:35.708526    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf" podUID="f5138a30-bf88-4515-a33f-6d955d4507b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 17:44:44.705151 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:44:44.705113    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb" podUID="b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 17:44:45.709086 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:44:45.709045    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf" podUID="f5138a30-bf88-4515-a33f-6d955d4507b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 17:44:54.705107 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:44:54.705066    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb" podUID="b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 17:44:55.708379 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:44:55.708339    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf" podUID="f5138a30-bf88-4515-a33f-6d955d4507b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 17:44:56.285453 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:44:56.285408    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb" podUID="b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 17:45:01.782573 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:01.782537    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:05.709153 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:05.709079    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"
Jul 28 17:45:06.289771 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:06.289744    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"
Jul 28 17:45:12.403323 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:12.403291    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"]
Jul 28 17:45:12.403839 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:12.403564    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf" podUID="f5138a30-bf88-4515-a33f-6d955d4507b8" containerName="kserve-container" containerID="cri-o://f944d1a7b416baac43fb538aac2e35ccbaf132f34c106893f4629b8ab73894b3" gracePeriod=30
Jul 28 17:45:12.404328 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:12.403907    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf" podUID="f5138a30-bf88-4515-a33f-6d955d4507b8" containerName="kube-rbac-proxy" containerID="cri-o://9a27f656f0c2af8adf44e4a76a2b5c0d799c50bf37684f17643c2000a346e2a4" gracePeriod=30
Jul 28 17:45:12.419134 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:12.419110    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:12.431378 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:12.431354    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"]
Jul 28 17:45:12.431619 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:12.431594    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb" podUID="b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" containerName="kserve-container" containerID="cri-o://4a66addba5adcb920b69d9412a50a2c2a3b84dcc9069045b4fb2f8d39ca67164" gracePeriod=30
Jul 28 17:45:12.431715 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:12.431633    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb" podUID="b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" containerName="kube-rbac-proxy" containerID="cri-o://5a2649206414963594d67723a2a7f847e0d7dc952f43ecef71c4554f2548f92d" gracePeriod=30
Jul 28 17:45:12.474220 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:12.474194    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:12.998481 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:12.998448    2581 generic.go:356] "Generic (PLEG): container finished" podID="b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" containerID="5a2649206414963594d67723a2a7f847e0d7dc952f43ecef71c4554f2548f92d" exitCode=2
Jul 28 17:45:12.998664 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:12.998520    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb" event={"ID":"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd","Type":"ContainerDied","Data":"5a2649206414963594d67723a2a7f847e0d7dc952f43ecef71c4554f2548f92d"}
Jul 28 17:45:13.000538 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:13.000516    2581 generic.go:356] "Generic (PLEG): container finished" podID="f5138a30-bf88-4515-a33f-6d955d4507b8" containerID="9a27f656f0c2af8adf44e4a76a2b5c0d799c50bf37684f17643c2000a346e2a4" exitCode=2
Jul 28 17:45:13.000669 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:13.000579    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf" event={"ID":"f5138a30-bf88-4515-a33f-6d955d4507b8","Type":"ContainerDied","Data":"9a27f656f0c2af8adf44e4a76a2b5c0d799c50bf37684f17643c2000a346e2a4"}
Jul 28 17:45:14.699834 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:14.699788    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb" podUID="b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Jul 28 17:45:15.703276 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:15.703225    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf" podUID="f5138a30-bf88-4515-a33f-6d955d4507b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 28 17:45:15.708597 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:15.708567    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf" podUID="f5138a30-bf88-4515-a33f-6d955d4507b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 17:45:16.285769 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:16.285721    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb" podUID="b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 17:45:17.286488 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.286459    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"
Jul 28 17:45:17.289241 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.289218    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"
Jul 28 17:45:17.344704 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.344613    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5138a30-bf88-4515-a33f-6d955d4507b8-kserve-provision-location\") pod \"f5138a30-bf88-4515-a33f-6d955d4507b8\" (UID: \"f5138a30-bf88-4515-a33f-6d955d4507b8\") "
Jul 28 17:45:17.344704 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.344657    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5138a30-bf88-4515-a33f-6d955d4507b8-kube-rbac-proxy-sar-config\") pod \"f5138a30-bf88-4515-a33f-6d955d4507b8\" (UID: \"f5138a30-bf88-4515-a33f-6d955d4507b8\") "
Jul 28 17:45:17.344704 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.344679    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h9md\" (UniqueName: \"kubernetes.io/projected/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-kube-api-access-2h9md\") pod \"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd\" (UID: \"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd\") "
Jul 28 17:45:17.345007 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.344709    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5138a30-bf88-4515-a33f-6d955d4507b8-proxy-tls\") pod \"f5138a30-bf88-4515-a33f-6d955d4507b8\" (UID: \"f5138a30-bf88-4515-a33f-6d955d4507b8\") "
Jul 28 17:45:17.345007 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.344733    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-kube-rbac-proxy-sar-config\") pod \"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd\" (UID: \"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd\") "
Jul 28 17:45:17.345007 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.344769    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-cabundle-cert\") pod \"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd\" (UID: \"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd\") "
Jul 28 17:45:17.345007 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.344796    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-kserve-provision-location\") pod \"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd\" (UID: \"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd\") "
Jul 28 17:45:17.345007 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.344826    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnkmd\" (UniqueName: \"kubernetes.io/projected/f5138a30-bf88-4515-a33f-6d955d4507b8-kube-api-access-dnkmd\") pod \"f5138a30-bf88-4515-a33f-6d955d4507b8\" (UID: \"f5138a30-bf88-4515-a33f-6d955d4507b8\") "
Jul 28 17:45:17.345007 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.344870    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5138a30-bf88-4515-a33f-6d955d4507b8-cabundle-cert\") pod \"f5138a30-bf88-4515-a33f-6d955d4507b8\" (UID: \"f5138a30-bf88-4515-a33f-6d955d4507b8\") "
Jul 28 17:45:17.345007 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.344904    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-proxy-tls\") pod \"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd\" (UID: \"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd\") "
Jul 28 17:45:17.345365 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.345008    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5138a30-bf88-4515-a33f-6d955d4507b8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f5138a30-bf88-4515-a33f-6d955d4507b8" (UID: "f5138a30-bf88-4515-a33f-6d955d4507b8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:45:17.345365 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.345047    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5138a30-bf88-4515-a33f-6d955d4507b8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f5138a30-bf88-4515-a33f-6d955d4507b8" (UID: "f5138a30-bf88-4515-a33f-6d955d4507b8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:45:17.345365 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.345241    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5138a30-bf88-4515-a33f-6d955d4507b8-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:45:17.345365 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.345263    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5138a30-bf88-4515-a33f-6d955d4507b8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:45:17.345580 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.345484    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" (UID: "b0eeff6d-1e9a-4d51-a705-b6a059ce6acd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:45:17.345580 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.345527    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" (UID: "b0eeff6d-1e9a-4d51-a705-b6a059ce6acd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:45:17.345580 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.345547    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5138a30-bf88-4515-a33f-6d955d4507b8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f5138a30-bf88-4515-a33f-6d955d4507b8" (UID: "f5138a30-bf88-4515-a33f-6d955d4507b8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:45:17.345738 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.345668    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" (UID: "b0eeff6d-1e9a-4d51-a705-b6a059ce6acd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:45:17.347237 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.347213    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5138a30-bf88-4515-a33f-6d955d4507b8-kube-api-access-dnkmd" (OuterVolumeSpecName: "kube-api-access-dnkmd") pod "f5138a30-bf88-4515-a33f-6d955d4507b8" (UID: "f5138a30-bf88-4515-a33f-6d955d4507b8"). InnerVolumeSpecName "kube-api-access-dnkmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:45:17.347426 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.347406    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5138a30-bf88-4515-a33f-6d955d4507b8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f5138a30-bf88-4515-a33f-6d955d4507b8" (UID: "f5138a30-bf88-4515-a33f-6d955d4507b8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:45:17.347426 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.347410    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" (UID: "b0eeff6d-1e9a-4d51-a705-b6a059ce6acd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:45:17.347558 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.347439    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-kube-api-access-2h9md" (OuterVolumeSpecName: "kube-api-access-2h9md") pod "b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" (UID: "b0eeff6d-1e9a-4d51-a705-b6a059ce6acd"). InnerVolumeSpecName "kube-api-access-2h9md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:45:17.445876 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.445840    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5138a30-bf88-4515-a33f-6d955d4507b8-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:45:17.445876 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.445872    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:45:17.445876 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.445882    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:45:17.446142 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.445891    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:45:17.446142 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.445899    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnkmd\" (UniqueName: \"kubernetes.io/projected/f5138a30-bf88-4515-a33f-6d955d4507b8-kube-api-access-dnkmd\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:45:17.446142 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.445910    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5138a30-bf88-4515-a33f-6d955d4507b8-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:45:17.446142 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.445918    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:45:17.446142 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:17.445925    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h9md\" (UniqueName: \"kubernetes.io/projected/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd-kube-api-access-2h9md\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:45:18.019902 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.019861    2581 generic.go:356] "Generic (PLEG): container finished" podID="b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" containerID="4a66addba5adcb920b69d9412a50a2c2a3b84dcc9069045b4fb2f8d39ca67164" exitCode=0
Jul 28 17:45:18.020095 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.019921    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"
Jul 28 17:45:18.020095 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.019956    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb" event={"ID":"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd","Type":"ContainerDied","Data":"4a66addba5adcb920b69d9412a50a2c2a3b84dcc9069045b4fb2f8d39ca67164"}
Jul 28 17:45:18.020095 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.020020    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb" event={"ID":"b0eeff6d-1e9a-4d51-a705-b6a059ce6acd","Type":"ContainerDied","Data":"abe4b2bc1f1b43b6f70f08ebc2578e99603973549d4f772fbcf13eea92ea3602"}
Jul 28 17:45:18.020095 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.020060    2581 scope.go:117] "RemoveContainer" containerID="5a2649206414963594d67723a2a7f847e0d7dc952f43ecef71c4554f2548f92d"
Jul 28 17:45:18.022070 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.022045    2581 generic.go:356] "Generic (PLEG): container finished" podID="f5138a30-bf88-4515-a33f-6d955d4507b8" containerID="f944d1a7b416baac43fb538aac2e35ccbaf132f34c106893f4629b8ab73894b3" exitCode=0
Jul 28 17:45:18.022166 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.022088    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf" event={"ID":"f5138a30-bf88-4515-a33f-6d955d4507b8","Type":"ContainerDied","Data":"f944d1a7b416baac43fb538aac2e35ccbaf132f34c106893f4629b8ab73894b3"}
Jul 28 17:45:18.022166 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.022101    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"
Jul 28 17:45:18.022166 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.022138    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf" event={"ID":"f5138a30-bf88-4515-a33f-6d955d4507b8","Type":"ContainerDied","Data":"4830e5a211e1a20c4937eb181a9a22998c73c3e58747f6b7b011c6bfff30f28e"}
Jul 28 17:45:18.028353 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.028326    2581 scope.go:117] "RemoveContainer" containerID="4a66addba5adcb920b69d9412a50a2c2a3b84dcc9069045b4fb2f8d39ca67164"
Jul 28 17:45:18.035632 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.035614    2581 scope.go:117] "RemoveContainer" containerID="42bf868e64d50c897b328e18d97ecfa66c90c463e57bad181f747a4569743f91"
Jul 28 17:45:18.042838 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.042820    2581 scope.go:117] "RemoveContainer" containerID="5a2649206414963594d67723a2a7f847e0d7dc952f43ecef71c4554f2548f92d"
Jul 28 17:45:18.043105 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:45:18.043084    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a2649206414963594d67723a2a7f847e0d7dc952f43ecef71c4554f2548f92d\": container with ID starting with 5a2649206414963594d67723a2a7f847e0d7dc952f43ecef71c4554f2548f92d not found: ID does not exist" containerID="5a2649206414963594d67723a2a7f847e0d7dc952f43ecef71c4554f2548f92d"
Jul 28 17:45:18.043162 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.043112    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a2649206414963594d67723a2a7f847e0d7dc952f43ecef71c4554f2548f92d"} err="failed to get container status \"5a2649206414963594d67723a2a7f847e0d7dc952f43ecef71c4554f2548f92d\": rpc error: code = NotFound desc = could not find container \"5a2649206414963594d67723a2a7f847e0d7dc952f43ecef71c4554f2548f92d\": container with ID starting with 5a2649206414963594d67723a2a7f847e0d7dc952f43ecef71c4554f2548f92d not found: ID does not exist"
Jul 28 17:45:18.043162 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.043130    2581 scope.go:117] "RemoveContainer" containerID="4a66addba5adcb920b69d9412a50a2c2a3b84dcc9069045b4fb2f8d39ca67164"
Jul 28 17:45:18.043350 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:45:18.043333    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a66addba5adcb920b69d9412a50a2c2a3b84dcc9069045b4fb2f8d39ca67164\": container with ID starting with 4a66addba5adcb920b69d9412a50a2c2a3b84dcc9069045b4fb2f8d39ca67164 not found: ID does not exist" containerID="4a66addba5adcb920b69d9412a50a2c2a3b84dcc9069045b4fb2f8d39ca67164"
Jul 28 17:45:18.043394 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.043357    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a66addba5adcb920b69d9412a50a2c2a3b84dcc9069045b4fb2f8d39ca67164"} err="failed to get container status \"4a66addba5adcb920b69d9412a50a2c2a3b84dcc9069045b4fb2f8d39ca67164\": rpc error: code = NotFound desc = could not find container \"4a66addba5adcb920b69d9412a50a2c2a3b84dcc9069045b4fb2f8d39ca67164\": container with ID starting with 4a66addba5adcb920b69d9412a50a2c2a3b84dcc9069045b4fb2f8d39ca67164 not found: ID does not exist"
Jul 28 17:45:18.043394 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.043373    2581 scope.go:117] "RemoveContainer" containerID="42bf868e64d50c897b328e18d97ecfa66c90c463e57bad181f747a4569743f91"
Jul 28 17:45:18.043588 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:45:18.043567    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42bf868e64d50c897b328e18d97ecfa66c90c463e57bad181f747a4569743f91\": container with ID starting with 42bf868e64d50c897b328e18d97ecfa66c90c463e57bad181f747a4569743f91 not found: ID does not exist" containerID="42bf868e64d50c897b328e18d97ecfa66c90c463e57bad181f747a4569743f91"
Jul 28 17:45:18.043627 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.043594    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42bf868e64d50c897b328e18d97ecfa66c90c463e57bad181f747a4569743f91"} err="failed to get container status \"42bf868e64d50c897b328e18d97ecfa66c90c463e57bad181f747a4569743f91\": rpc error: code = NotFound desc = could not find container \"42bf868e64d50c897b328e18d97ecfa66c90c463e57bad181f747a4569743f91\": container with ID starting with 42bf868e64d50c897b328e18d97ecfa66c90c463e57bad181f747a4569743f91 not found: ID does not exist"
Jul 28 17:45:18.043627 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.043607    2581 scope.go:117] "RemoveContainer" containerID="9a27f656f0c2af8adf44e4a76a2b5c0d799c50bf37684f17643c2000a346e2a4"
Jul 28 17:45:18.047222 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.047201    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"]
Jul 28 17:45:18.050731 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.050715    2581 scope.go:117] "RemoveContainer" containerID="f944d1a7b416baac43fb538aac2e35ccbaf132f34c106893f4629b8ab73894b3"
Jul 28 17:45:18.050865 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.050840    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6c87f69f44-xwkkb"]
Jul 28 17:45:18.057574 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.057548    2581 scope.go:117] "RemoveContainer" containerID="37646a2dba678a97e09c847d39e1f849c51f7f7f654c8cbaa826a2fc06a1858b"
Jul 28 17:45:18.063735 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.063711    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"]
Jul 28 17:45:18.064930 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.064909    2581 scope.go:117] "RemoveContainer" containerID="9a27f656f0c2af8adf44e4a76a2b5c0d799c50bf37684f17643c2000a346e2a4"
Jul 28 17:45:18.065240 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:45:18.065220    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a27f656f0c2af8adf44e4a76a2b5c0d799c50bf37684f17643c2000a346e2a4\": container with ID starting with 9a27f656f0c2af8adf44e4a76a2b5c0d799c50bf37684f17643c2000a346e2a4 not found: ID does not exist" containerID="9a27f656f0c2af8adf44e4a76a2b5c0d799c50bf37684f17643c2000a346e2a4"
Jul 28 17:45:18.065337 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.065249    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a27f656f0c2af8adf44e4a76a2b5c0d799c50bf37684f17643c2000a346e2a4"} err="failed to get container status \"9a27f656f0c2af8adf44e4a76a2b5c0d799c50bf37684f17643c2000a346e2a4\": rpc error: code = NotFound desc = could not find container \"9a27f656f0c2af8adf44e4a76a2b5c0d799c50bf37684f17643c2000a346e2a4\": container with ID starting with 9a27f656f0c2af8adf44e4a76a2b5c0d799c50bf37684f17643c2000a346e2a4 not found: ID does not exist"
Jul 28 17:45:18.065337 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.065273    2581 scope.go:117] "RemoveContainer" containerID="f944d1a7b416baac43fb538aac2e35ccbaf132f34c106893f4629b8ab73894b3"
Jul 28 17:45:18.065628 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:45:18.065605    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f944d1a7b416baac43fb538aac2e35ccbaf132f34c106893f4629b8ab73894b3\": container with ID starting with f944d1a7b416baac43fb538aac2e35ccbaf132f34c106893f4629b8ab73894b3 not found: ID does not exist" containerID="f944d1a7b416baac43fb538aac2e35ccbaf132f34c106893f4629b8ab73894b3"
Jul 28 17:45:18.065740 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.065634    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f944d1a7b416baac43fb538aac2e35ccbaf132f34c106893f4629b8ab73894b3"} err="failed to get container status \"f944d1a7b416baac43fb538aac2e35ccbaf132f34c106893f4629b8ab73894b3\": rpc error: code = NotFound desc = could not find container \"f944d1a7b416baac43fb538aac2e35ccbaf132f34c106893f4629b8ab73894b3\": container with ID starting with f944d1a7b416baac43fb538aac2e35ccbaf132f34c106893f4629b8ab73894b3 not found: ID does not exist"
Jul 28 17:45:18.065740 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.065657    2581 scope.go:117] "RemoveContainer" containerID="37646a2dba678a97e09c847d39e1f849c51f7f7f654c8cbaa826a2fc06a1858b"
Jul 28 17:45:18.065963 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:45:18.065929    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37646a2dba678a97e09c847d39e1f849c51f7f7f654c8cbaa826a2fc06a1858b\": container with ID starting with 37646a2dba678a97e09c847d39e1f849c51f7f7f654c8cbaa826a2fc06a1858b not found: ID does not exist" containerID="37646a2dba678a97e09c847d39e1f849c51f7f7f654c8cbaa826a2fc06a1858b"
Jul 28 17:45:18.066025 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.065967    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37646a2dba678a97e09c847d39e1f849c51f7f7f654c8cbaa826a2fc06a1858b"} err="failed to get container status \"37646a2dba678a97e09c847d39e1f849c51f7f7f654c8cbaa826a2fc06a1858b\": rpc error: code = NotFound desc = could not find container \"37646a2dba678a97e09c847d39e1f849c51f7f7f654c8cbaa826a2fc06a1858b\": container with ID starting with 37646a2dba678a97e09c847d39e1f849c51f7f7f654c8cbaa826a2fc06a1858b not found: ID does not exist"
Jul 28 17:45:18.066829 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.066809    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-44hnf"]
Jul 28 17:45:18.289393 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.289315    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" path="/var/lib/kubelet/pods/b0eeff6d-1e9a-4d51-a705-b6a059ce6acd/volumes"
Jul 28 17:45:18.289792 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:18.289779    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5138a30-bf88-4515-a33f-6d955d4507b8" path="/var/lib/kubelet/pods/f5138a30-bf88-4515-a33f-6d955d4507b8/volumes"
Jul 28 17:45:22.725023 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.724984    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:22.826824 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.826785    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"]
Jul 28 17:45:22.827050 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.827037    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" containerName="kube-rbac-proxy"
Jul 28 17:45:22.827050 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.827050    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" containerName="kube-rbac-proxy"
Jul 28 17:45:22.827174 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.827063    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" containerName="storage-initializer"
Jul 28 17:45:22.827174 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.827068    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" containerName="storage-initializer"
Jul 28 17:45:22.827174 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.827076    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" containerName="kserve-container"
Jul 28 17:45:22.827174 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.827082    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" containerName="kserve-container"
Jul 28 17:45:22.827174 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.827089    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5138a30-bf88-4515-a33f-6d955d4507b8" containerName="storage-initializer"
Jul 28 17:45:22.827174 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.827094    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5138a30-bf88-4515-a33f-6d955d4507b8" containerName="storage-initializer"
Jul 28 17:45:22.827174 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.827102    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5138a30-bf88-4515-a33f-6d955d4507b8" containerName="kube-rbac-proxy"
Jul 28 17:45:22.827174 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.827107    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5138a30-bf88-4515-a33f-6d955d4507b8" containerName="kube-rbac-proxy"
Jul 28 17:45:22.827174 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.827116    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5138a30-bf88-4515-a33f-6d955d4507b8" containerName="kserve-container"
Jul 28 17:45:22.827174 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.827121    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5138a30-bf88-4515-a33f-6d955d4507b8" containerName="kserve-container"
Jul 28 17:45:22.827174 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.827161    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" containerName="kserve-container"
Jul 28 17:45:22.827174 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.827169    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5138a30-bf88-4515-a33f-6d955d4507b8" containerName="kube-rbac-proxy"
Jul 28 17:45:22.827174 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.827176    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0eeff6d-1e9a-4d51-a705-b6a059ce6acd" containerName="kube-rbac-proxy"
Jul 28 17:45:22.827174 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.827182    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5138a30-bf88-4515-a33f-6d955d4507b8" containerName="kserve-container"
Jul 28 17:45:22.831213 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.831194    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"
Jul 28 17:45:22.834570 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.834550    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 17:45:22.834853 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.834825    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:45:22.834995 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.834866    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:45:22.835063 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.834994    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Jul 28 17:45:22.835808 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.835792    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:45:22.835860 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.835831    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Jul 28 17:45:22.835898 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.835862    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:45:22.844698 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.844677    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"]
Jul 28 17:45:22.883660 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.883637    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b8eaa36-c531-4b81-b165-4715081b1c2e-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-xppg6\" (UID: \"1b8eaa36-c531-4b81-b165-4715081b1c2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"
Jul 28 17:45:22.883805 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.883677    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b8eaa36-c531-4b81-b165-4715081b1c2e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-xppg6\" (UID: \"1b8eaa36-c531-4b81-b165-4715081b1c2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"
Jul 28 17:45:22.883805 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.883704    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b8eaa36-c531-4b81-b165-4715081b1c2e-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-xppg6\" (UID: \"1b8eaa36-c531-4b81-b165-4715081b1c2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"
Jul 28 17:45:22.883805 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.883777    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv6g5\" (UniqueName: \"kubernetes.io/projected/1b8eaa36-c531-4b81-b165-4715081b1c2e-kube-api-access-lv6g5\") pod \"isvc-lightgbm-predictor-54f85ddf88-xppg6\" (UID: \"1b8eaa36-c531-4b81-b165-4715081b1c2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"
Jul 28 17:45:22.883992 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.883818    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b8eaa36-c531-4b81-b165-4715081b1c2e-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-xppg6\" (UID: \"1b8eaa36-c531-4b81-b165-4715081b1c2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"
Jul 28 17:45:22.985102 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.985019    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b8eaa36-c531-4b81-b165-4715081b1c2e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-xppg6\" (UID: \"1b8eaa36-c531-4b81-b165-4715081b1c2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"
Jul 28 17:45:22.985102 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.985061    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b8eaa36-c531-4b81-b165-4715081b1c2e-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-xppg6\" (UID: \"1b8eaa36-c531-4b81-b165-4715081b1c2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"
Jul 28 17:45:22.985102 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.985090    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv6g5\" (UniqueName: \"kubernetes.io/projected/1b8eaa36-c531-4b81-b165-4715081b1c2e-kube-api-access-lv6g5\") pod \"isvc-lightgbm-predictor-54f85ddf88-xppg6\" (UID: \"1b8eaa36-c531-4b81-b165-4715081b1c2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"
Jul 28 17:45:22.985331 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.985129    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b8eaa36-c531-4b81-b165-4715081b1c2e-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-xppg6\" (UID: \"1b8eaa36-c531-4b81-b165-4715081b1c2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"
Jul 28 17:45:22.985331 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.985151    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b8eaa36-c531-4b81-b165-4715081b1c2e-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-xppg6\" (UID: \"1b8eaa36-c531-4b81-b165-4715081b1c2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"
Jul 28 17:45:22.985580 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.985555    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b8eaa36-c531-4b81-b165-4715081b1c2e-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-xppg6\" (UID: \"1b8eaa36-c531-4b81-b165-4715081b1c2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"
Jul 28 17:45:22.985768 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.985750    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b8eaa36-c531-4b81-b165-4715081b1c2e-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-xppg6\" (UID: \"1b8eaa36-c531-4b81-b165-4715081b1c2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"
Jul 28 17:45:22.985814 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.985774    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b8eaa36-c531-4b81-b165-4715081b1c2e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-xppg6\" (UID: \"1b8eaa36-c531-4b81-b165-4715081b1c2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"
Jul 28 17:45:22.987600 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.987578    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b8eaa36-c531-4b81-b165-4715081b1c2e-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-xppg6\" (UID: \"1b8eaa36-c531-4b81-b165-4715081b1c2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"
Jul 28 17:45:22.996383 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:22.996357    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv6g5\" (UniqueName: \"kubernetes.io/projected/1b8eaa36-c531-4b81-b165-4715081b1c2e-kube-api-access-lv6g5\") pod \"isvc-lightgbm-predictor-54f85ddf88-xppg6\" (UID: \"1b8eaa36-c531-4b81-b165-4715081b1c2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"
Jul 28 17:45:23.011417 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:23.011391    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:45:23.141082 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:23.141044    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"
Jul 28 17:45:23.278988 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:23.278960    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"]
Jul 28 17:45:23.279502 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:45:23.279480    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8eaa36_c531_4b81_b165_4715081b1c2e.slice/crio-9c529d32617e2c807f4a916b865f3e514552c9ed1a39d034736012b19aefcf28 WatchSource:0}: Error finding container 9c529d32617e2c807f4a916b865f3e514552c9ed1a39d034736012b19aefcf28: Status 404 returned error can't find the container with id 9c529d32617e2c807f4a916b865f3e514552c9ed1a39d034736012b19aefcf28
Jul 28 17:45:24.045234 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:24.045200    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6" event={"ID":"1b8eaa36-c531-4b81-b165-4715081b1c2e","Type":"ContainerStarted","Data":"a1b20c2037522cef6c1c7433f960255c0137206bf876480f9d2eaf7a3b936a69"}
Jul 28 17:45:24.045622 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:24.045259    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6" event={"ID":"1b8eaa36-c531-4b81-b165-4715081b1c2e","Type":"ContainerStarted","Data":"9c529d32617e2c807f4a916b865f3e514552c9ed1a39d034736012b19aefcf28"}
Jul 28 17:45:28.061070 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:28.061033    2581 generic.go:356] "Generic (PLEG): container finished" podID="1b8eaa36-c531-4b81-b165-4715081b1c2e" containerID="a1b20c2037522cef6c1c7433f960255c0137206bf876480f9d2eaf7a3b936a69" exitCode=0
Jul 28 17:45:28.061070 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:28.061071    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6" event={"ID":"1b8eaa36-c531-4b81-b165-4715081b1c2e","Type":"ContainerDied","Data":"a1b20c2037522cef6c1c7433f960255c0137206bf876480f9d2eaf7a3b936a69"}
Jul 28 17:45:28.062197 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:28.062180    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:45:36.094081 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:36.094047    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6" event={"ID":"1b8eaa36-c531-4b81-b165-4715081b1c2e","Type":"ContainerStarted","Data":"00c4f9ae7fcd6d1963403eac026ea1706c075d64e0fd0a14414acc30e9787ff3"}
Jul 28 17:45:36.094486 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:36.094107    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6" event={"ID":"1b8eaa36-c531-4b81-b165-4715081b1c2e","Type":"ContainerStarted","Data":"bd28ec96c660ea96e170b39a62a2f9cec003a47cd320484b556e058b15bfbc6c"}
Jul 28 17:45:36.094486 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:36.094433    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"
Jul 28 17:45:36.094486 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:36.094484    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"
Jul 28 17:45:36.095802 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:36.095772    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6" podUID="1b8eaa36-c531-4b81-b165-4715081b1c2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 17:45:36.124565 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:36.124522    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6" podStartSLOduration=3.0711194 podStartE2EDuration="14.124509219s" totalImagesPullingTime="7.010086515s" totalInitContainerRuntime="4.043303304s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:45:36.123919469 +0000 UTC m=+2184.398535901" watchObservedRunningTime="2026-07-28 17:45:36.124509219 +0000 UTC m=+2184.399125651"
Jul 28 17:45:37.097344 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:37.097301    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6" podUID="1b8eaa36-c531-4b81-b165-4715081b1c2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 17:45:42.102321 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:42.102290    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"
Jul 28 17:45:42.102780 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:42.102752    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6" podUID="1b8eaa36-c531-4b81-b165-4715081b1c2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 17:45:52.103576 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:45:52.103537    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6" podUID="1b8eaa36-c531-4b81-b165-4715081b1c2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 17:46:02.103162 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:02.103121    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6" podUID="1b8eaa36-c531-4b81-b165-4715081b1c2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 17:46:12.103168 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:12.103117    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6" podUID="1b8eaa36-c531-4b81-b165-4715081b1c2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 17:46:22.103062 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:22.103022    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6" podUID="1b8eaa36-c531-4b81-b165-4715081b1c2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 17:46:32.103542 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:32.103500    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6" podUID="1b8eaa36-c531-4b81-b165-4715081b1c2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 17:46:42.103392 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:42.103306    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6" podUID="1b8eaa36-c531-4b81-b165-4715081b1c2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 17:46:49.286090 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:49.286051    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"
Jul 28 17:46:52.880001 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:52.879968    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"]
Jul 28 17:46:52.880386 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:52.880181    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6" podUID="1b8eaa36-c531-4b81-b165-4715081b1c2e" containerName="kserve-container" containerID="cri-o://bd28ec96c660ea96e170b39a62a2f9cec003a47cd320484b556e058b15bfbc6c" gracePeriod=30
Jul 28 17:46:52.880386 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:52.880221    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6" podUID="1b8eaa36-c531-4b81-b165-4715081b1c2e" containerName="kube-rbac-proxy" containerID="cri-o://00c4f9ae7fcd6d1963403eac026ea1706c075d64e0fd0a14414acc30e9787ff3" gracePeriod=30
Jul 28 17:46:52.906141 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:52.906113    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:53.369041 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:53.369002    2581 generic.go:356] "Generic (PLEG): container finished" podID="1b8eaa36-c531-4b81-b165-4715081b1c2e" containerID="00c4f9ae7fcd6d1963403eac026ea1706c075d64e0fd0a14414acc30e9787ff3" exitCode=2
Jul 28 17:46:53.369205 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:53.369074    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6" event={"ID":"1b8eaa36-c531-4b81-b165-4715081b1c2e","Type":"ContainerDied","Data":"00c4f9ae7fcd6d1963403eac026ea1706c075d64e0fd0a14414acc30e9787ff3"}
Jul 28 17:46:57.018959 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.018919    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"
Jul 28 17:46:57.040430 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.040405    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b8eaa36-c531-4b81-b165-4715081b1c2e-kserve-provision-location\") pod \"1b8eaa36-c531-4b81-b165-4715081b1c2e\" (UID: \"1b8eaa36-c531-4b81-b165-4715081b1c2e\") "
Jul 28 17:46:57.040523 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.040435    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lv6g5\" (UniqueName: \"kubernetes.io/projected/1b8eaa36-c531-4b81-b165-4715081b1c2e-kube-api-access-lv6g5\") pod \"1b8eaa36-c531-4b81-b165-4715081b1c2e\" (UID: \"1b8eaa36-c531-4b81-b165-4715081b1c2e\") "
Jul 28 17:46:57.040523 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.040458    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b8eaa36-c531-4b81-b165-4715081b1c2e-kube-rbac-proxy-sar-config\") pod \"1b8eaa36-c531-4b81-b165-4715081b1c2e\" (UID: \"1b8eaa36-c531-4b81-b165-4715081b1c2e\") "
Jul 28 17:46:57.040523 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.040511    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b8eaa36-c531-4b81-b165-4715081b1c2e-cabundle-cert\") pod \"1b8eaa36-c531-4b81-b165-4715081b1c2e\" (UID: \"1b8eaa36-c531-4b81-b165-4715081b1c2e\") "
Jul 28 17:46:57.040676 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.040562    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b8eaa36-c531-4b81-b165-4715081b1c2e-proxy-tls\") pod \"1b8eaa36-c531-4b81-b165-4715081b1c2e\" (UID: \"1b8eaa36-c531-4b81-b165-4715081b1c2e\") "
Jul 28 17:46:57.040832 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.040802    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8eaa36-c531-4b81-b165-4715081b1c2e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1b8eaa36-c531-4b81-b165-4715081b1c2e" (UID: "1b8eaa36-c531-4b81-b165-4715081b1c2e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:46:57.040916 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.040880    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b8eaa36-c531-4b81-b165-4715081b1c2e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1b8eaa36-c531-4b81-b165-4715081b1c2e" (UID: "1b8eaa36-c531-4b81-b165-4715081b1c2e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:46:57.040916 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.040894    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b8eaa36-c531-4b81-b165-4715081b1c2e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1b8eaa36-c531-4b81-b165-4715081b1c2e" (UID: "1b8eaa36-c531-4b81-b165-4715081b1c2e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:46:57.042407 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.042387    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b8eaa36-c531-4b81-b165-4715081b1c2e-kube-api-access-lv6g5" (OuterVolumeSpecName: "kube-api-access-lv6g5") pod "1b8eaa36-c531-4b81-b165-4715081b1c2e" (UID: "1b8eaa36-c531-4b81-b165-4715081b1c2e"). InnerVolumeSpecName "kube-api-access-lv6g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:46:57.042733 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.042704    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b8eaa36-c531-4b81-b165-4715081b1c2e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1b8eaa36-c531-4b81-b165-4715081b1c2e" (UID: "1b8eaa36-c531-4b81-b165-4715081b1c2e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:46:57.141640 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.141578    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b8eaa36-c531-4b81-b165-4715081b1c2e-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:46:57.141640 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.141599    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv6g5\" (UniqueName: \"kubernetes.io/projected/1b8eaa36-c531-4b81-b165-4715081b1c2e-kube-api-access-lv6g5\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:46:57.141640 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.141611    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b8eaa36-c531-4b81-b165-4715081b1c2e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:46:57.141640 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.141619    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b8eaa36-c531-4b81-b165-4715081b1c2e-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:46:57.141640 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.141628    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b8eaa36-c531-4b81-b165-4715081b1c2e-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:46:57.387274 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.387242    2581 generic.go:356] "Generic (PLEG): container finished" podID="1b8eaa36-c531-4b81-b165-4715081b1c2e" containerID="bd28ec96c660ea96e170b39a62a2f9cec003a47cd320484b556e058b15bfbc6c" exitCode=0
Jul 28 17:46:57.387463 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.387319    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"
Jul 28 17:46:57.387463 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.387324    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6" event={"ID":"1b8eaa36-c531-4b81-b165-4715081b1c2e","Type":"ContainerDied","Data":"bd28ec96c660ea96e170b39a62a2f9cec003a47cd320484b556e058b15bfbc6c"}
Jul 28 17:46:57.387463 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.387383    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6" event={"ID":"1b8eaa36-c531-4b81-b165-4715081b1c2e","Type":"ContainerDied","Data":"9c529d32617e2c807f4a916b865f3e514552c9ed1a39d034736012b19aefcf28"}
Jul 28 17:46:57.387463 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.387412    2581 scope.go:117] "RemoveContainer" containerID="00c4f9ae7fcd6d1963403eac026ea1706c075d64e0fd0a14414acc30e9787ff3"
Jul 28 17:46:57.395687 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.395672    2581 scope.go:117] "RemoveContainer" containerID="bd28ec96c660ea96e170b39a62a2f9cec003a47cd320484b556e058b15bfbc6c"
Jul 28 17:46:57.402712 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.402697    2581 scope.go:117] "RemoveContainer" containerID="a1b20c2037522cef6c1c7433f960255c0137206bf876480f9d2eaf7a3b936a69"
Jul 28 17:46:57.409177 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.409160    2581 scope.go:117] "RemoveContainer" containerID="00c4f9ae7fcd6d1963403eac026ea1706c075d64e0fd0a14414acc30e9787ff3"
Jul 28 17:46:57.409410 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:46:57.409393    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00c4f9ae7fcd6d1963403eac026ea1706c075d64e0fd0a14414acc30e9787ff3\": container with ID starting with 00c4f9ae7fcd6d1963403eac026ea1706c075d64e0fd0a14414acc30e9787ff3 not found: ID does not exist" containerID="00c4f9ae7fcd6d1963403eac026ea1706c075d64e0fd0a14414acc30e9787ff3"
Jul 28 17:46:57.409453 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.409419    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00c4f9ae7fcd6d1963403eac026ea1706c075d64e0fd0a14414acc30e9787ff3"} err="failed to get container status \"00c4f9ae7fcd6d1963403eac026ea1706c075d64e0fd0a14414acc30e9787ff3\": rpc error: code = NotFound desc = could not find container \"00c4f9ae7fcd6d1963403eac026ea1706c075d64e0fd0a14414acc30e9787ff3\": container with ID starting with 00c4f9ae7fcd6d1963403eac026ea1706c075d64e0fd0a14414acc30e9787ff3 not found: ID does not exist"
Jul 28 17:46:57.409453 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.409435    2581 scope.go:117] "RemoveContainer" containerID="bd28ec96c660ea96e170b39a62a2f9cec003a47cd320484b556e058b15bfbc6c"
Jul 28 17:46:57.409663 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:46:57.409647    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd28ec96c660ea96e170b39a62a2f9cec003a47cd320484b556e058b15bfbc6c\": container with ID starting with bd28ec96c660ea96e170b39a62a2f9cec003a47cd320484b556e058b15bfbc6c not found: ID does not exist" containerID="bd28ec96c660ea96e170b39a62a2f9cec003a47cd320484b556e058b15bfbc6c"
Jul 28 17:46:57.409713 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.409668    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd28ec96c660ea96e170b39a62a2f9cec003a47cd320484b556e058b15bfbc6c"} err="failed to get container status \"bd28ec96c660ea96e170b39a62a2f9cec003a47cd320484b556e058b15bfbc6c\": rpc error: code = NotFound desc = could not find container \"bd28ec96c660ea96e170b39a62a2f9cec003a47cd320484b556e058b15bfbc6c\": container with ID starting with bd28ec96c660ea96e170b39a62a2f9cec003a47cd320484b556e058b15bfbc6c not found: ID does not exist"
Jul 28 17:46:57.409713 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.409682    2581 scope.go:117] "RemoveContainer" containerID="a1b20c2037522cef6c1c7433f960255c0137206bf876480f9d2eaf7a3b936a69"
Jul 28 17:46:57.409860 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:46:57.409845    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1b20c2037522cef6c1c7433f960255c0137206bf876480f9d2eaf7a3b936a69\": container with ID starting with a1b20c2037522cef6c1c7433f960255c0137206bf876480f9d2eaf7a3b936a69 not found: ID does not exist" containerID="a1b20c2037522cef6c1c7433f960255c0137206bf876480f9d2eaf7a3b936a69"
Jul 28 17:46:57.409903 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.409862    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b20c2037522cef6c1c7433f960255c0137206bf876480f9d2eaf7a3b936a69"} err="failed to get container status \"a1b20c2037522cef6c1c7433f960255c0137206bf876480f9d2eaf7a3b936a69\": rpc error: code = NotFound desc = could not find container \"a1b20c2037522cef6c1c7433f960255c0137206bf876480f9d2eaf7a3b936a69\": container with ID starting with a1b20c2037522cef6c1c7433f960255c0137206bf876480f9d2eaf7a3b936a69 not found: ID does not exist"
Jul 28 17:46:57.414284 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.414264    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"]
Jul 28 17:46:57.419885 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:57.419863    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-xppg6"]
Jul 28 17:46:58.195734 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.195702    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"]
Jul 28 17:46:58.196144 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.195934    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b8eaa36-c531-4b81-b165-4715081b1c2e" containerName="storage-initializer"
Jul 28 17:46:58.196144 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.195962    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b8eaa36-c531-4b81-b165-4715081b1c2e" containerName="storage-initializer"
Jul 28 17:46:58.196144 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.195975    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b8eaa36-c531-4b81-b165-4715081b1c2e" containerName="kserve-container"
Jul 28 17:46:58.196144 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.195984    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b8eaa36-c531-4b81-b165-4715081b1c2e" containerName="kserve-container"
Jul 28 17:46:58.196144 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.195996    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b8eaa36-c531-4b81-b165-4715081b1c2e" containerName="kube-rbac-proxy"
Jul 28 17:46:58.196144 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.196003    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b8eaa36-c531-4b81-b165-4715081b1c2e" containerName="kube-rbac-proxy"
Jul 28 17:46:58.196144 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.196056    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b8eaa36-c531-4b81-b165-4715081b1c2e" containerName="kube-rbac-proxy"
Jul 28 17:46:58.196144 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.196066    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b8eaa36-c531-4b81-b165-4715081b1c2e" containerName="kserve-container"
Jul 28 17:46:58.200233 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.200216    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"
Jul 28 17:46:58.204617 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.204597    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:46:58.204742 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.204599    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:46:58.204742 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.204603    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 17:46:58.204742 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.204722    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Jul 28 17:46:58.205803 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.205787    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:46:58.205900 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.205885    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Jul 28 17:46:58.208380 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.208365    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:46:58.225586 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.225563    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"]
Jul 28 17:46:58.249170 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.249146    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb200891-0cae-432f-b1f1-6137e2a1f53b-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25\" (UID: \"eb200891-0cae-432f-b1f1-6137e2a1f53b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"
Jul 28 17:46:58.249262 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.249171    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb200891-0cae-432f-b1f1-6137e2a1f53b-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25\" (UID: \"eb200891-0cae-432f-b1f1-6137e2a1f53b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"
Jul 28 17:46:58.249262 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.249194    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8wbw\" (UniqueName: \"kubernetes.io/projected/eb200891-0cae-432f-b1f1-6137e2a1f53b-kube-api-access-w8wbw\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25\" (UID: \"eb200891-0cae-432f-b1f1-6137e2a1f53b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"
Jul 28 17:46:58.249349 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.249267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb200891-0cae-432f-b1f1-6137e2a1f53b-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25\" (UID: \"eb200891-0cae-432f-b1f1-6137e2a1f53b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"
Jul 28 17:46:58.249349 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.249308    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb200891-0cae-432f-b1f1-6137e2a1f53b-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25\" (UID: \"eb200891-0cae-432f-b1f1-6137e2a1f53b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"
Jul 28 17:46:58.290143 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.290116    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b8eaa36-c531-4b81-b165-4715081b1c2e" path="/var/lib/kubelet/pods/1b8eaa36-c531-4b81-b165-4715081b1c2e/volumes"
Jul 28 17:46:58.299127 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.299108    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:58.349831 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.349807    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb200891-0cae-432f-b1f1-6137e2a1f53b-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25\" (UID: \"eb200891-0cae-432f-b1f1-6137e2a1f53b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"
Jul 28 17:46:58.349915 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.349843    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb200891-0cae-432f-b1f1-6137e2a1f53b-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25\" (UID: \"eb200891-0cae-432f-b1f1-6137e2a1f53b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"
Jul 28 17:46:58.349915 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.349859    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb200891-0cae-432f-b1f1-6137e2a1f53b-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25\" (UID: \"eb200891-0cae-432f-b1f1-6137e2a1f53b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"
Jul 28 17:46:58.349915 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.349887    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8wbw\" (UniqueName: \"kubernetes.io/projected/eb200891-0cae-432f-b1f1-6137e2a1f53b-kube-api-access-w8wbw\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25\" (UID: \"eb200891-0cae-432f-b1f1-6137e2a1f53b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"
Jul 28 17:46:58.350067 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.349956    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb200891-0cae-432f-b1f1-6137e2a1f53b-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25\" (UID: \"eb200891-0cae-432f-b1f1-6137e2a1f53b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"
Jul 28 17:46:58.350452 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.350428    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb200891-0cae-432f-b1f1-6137e2a1f53b-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25\" (UID: \"eb200891-0cae-432f-b1f1-6137e2a1f53b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"
Jul 28 17:46:58.350611 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.350595    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb200891-0cae-432f-b1f1-6137e2a1f53b-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25\" (UID: \"eb200891-0cae-432f-b1f1-6137e2a1f53b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"
Jul 28 17:46:58.350668 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.350638    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb200891-0cae-432f-b1f1-6137e2a1f53b-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25\" (UID: \"eb200891-0cae-432f-b1f1-6137e2a1f53b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"
Jul 28 17:46:58.352165 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.352142    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb200891-0cae-432f-b1f1-6137e2a1f53b-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25\" (UID: \"eb200891-0cae-432f-b1f1-6137e2a1f53b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"
Jul 28 17:46:58.360495 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.360475    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8wbw\" (UniqueName: \"kubernetes.io/projected/eb200891-0cae-432f-b1f1-6137e2a1f53b-kube-api-access-w8wbw\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25\" (UID: \"eb200891-0cae-432f-b1f1-6137e2a1f53b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"
Jul 28 17:46:58.402976 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.402933    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:58.510419 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.510343    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"
Jul 28 17:46:58.640422 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.640400    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"]
Jul 28 17:46:58.642880 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:46:58.642839    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb200891_0cae_432f_b1f1_6137e2a1f53b.slice/crio-733763b52bf39038c76d75cba5a5a2756a29802cce42f882c1d2e864b47f37de WatchSource:0}: Error finding container 733763b52bf39038c76d75cba5a5a2756a29802cce42f882c1d2e864b47f37de: Status 404 returned error can't find the container with id 733763b52bf39038c76d75cba5a5a2756a29802cce42f882c1d2e864b47f37de
Jul 28 17:46:58.968422 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:58.968385    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:46:59.401736 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:59.401706    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25" event={"ID":"eb200891-0cae-432f-b1f1-6137e2a1f53b","Type":"ContainerStarted","Data":"3e16beafaf29d05631bd5898e7872c047f45ecf67cd68f266985c4faaff902f9"}
Jul 28 17:46:59.402104 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:46:59.401768    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25" event={"ID":"eb200891-0cae-432f-b1f1-6137e2a1f53b","Type":"ContainerStarted","Data":"733763b52bf39038c76d75cba5a5a2756a29802cce42f882c1d2e864b47f37de"}
Jul 28 17:47:03.416247 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:47:03.416210    2581 generic.go:356] "Generic (PLEG): container finished" podID="eb200891-0cae-432f-b1f1-6137e2a1f53b" containerID="3e16beafaf29d05631bd5898e7872c047f45ecf67cd68f266985c4faaff902f9" exitCode=0
Jul 28 17:47:03.416703 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:47:03.416288    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25" event={"ID":"eb200891-0cae-432f-b1f1-6137e2a1f53b","Type":"ContainerDied","Data":"3e16beafaf29d05631bd5898e7872c047f45ecf67cd68f266985c4faaff902f9"}
Jul 28 17:47:04.421612 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:47:04.421578    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25" event={"ID":"eb200891-0cae-432f-b1f1-6137e2a1f53b","Type":"ContainerStarted","Data":"cfc114537e9194bb2c431678c43c1894019f1988fb08d1bd59aabf495e7e091a"}
Jul 28 17:47:04.422064 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:47:04.421635    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25" event={"ID":"eb200891-0cae-432f-b1f1-6137e2a1f53b","Type":"ContainerStarted","Data":"8648d7468eba71765cbb9a8fabe56398b515311cca9a9a405a239d5ac30d08f0"}
Jul 28 17:47:04.422064 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:47:04.421911    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"
Jul 28 17:47:04.422186 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:47:04.422072    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"
Jul 28 17:47:04.423265 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:47:04.423241    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25" podUID="eb200891-0cae-432f-b1f1-6137e2a1f53b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 17:47:04.447149 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:47:04.447106    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25" podStartSLOduration=2.609860722 podStartE2EDuration="6.447090922s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.8372302s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:47:04.446095956 +0000 UTC m=+2272.720712389" watchObservedRunningTime="2026-07-28 17:47:04.447090922 +0000 UTC m=+2272.721707354"
Jul 28 17:47:05.424888 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:47:05.424851    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25" podUID="eb200891-0cae-432f-b1f1-6137e2a1f53b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 17:47:10.429275 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:47:10.429243    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"
Jul 28 17:47:10.429725 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:47:10.429698    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25" podUID="eb200891-0cae-432f-b1f1-6137e2a1f53b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 17:47:12.396704 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:47:12.396674    2581 scope.go:117] "RemoveContainer" containerID="ca3b0bffa1cc7704c3c8e77d1847b6a2afa7daa6988d23e184d72aea1af96720"
Jul 28 17:47:12.404001 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:47:12.403977    2581 scope.go:117] "RemoveContainer" containerID="2b37d799bfeb5f73ef01e483b5b19e8833e5b9c674c76bc0c914a916da3b74f5"
Jul 28 17:47:12.410505 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:47:12.410489    2581 scope.go:117] "RemoveContainer" containerID="4b1c3ec01891448c687a6dfb6084444f141def9ee7fc6a57db8744c9d4ff247d"
Jul 28 17:47:20.430214 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:47:20.430178    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25" podUID="eb200891-0cae-432f-b1f1-6137e2a1f53b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 17:47:30.221159 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:47:30.221122    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:30.323479 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:47:30.323451    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:47:30.430592 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:47:30.430560    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25" podUID="eb200891-0cae-432f-b1f1-6137e2a1f53b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 17:47:40.430543 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:47:40.430498    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25" podUID="eb200891-0cae-432f-b1f1-6137e2a1f53b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 17:47:50.430137 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:47:50.430097    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25" podUID="eb200891-0cae-432f-b1f1-6137e2a1f53b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 17:48:00.430491 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:00.430455    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25" podUID="eb200891-0cae-432f-b1f1-6137e2a1f53b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 17:48:10.433711 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:10.433624    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25" podUID="eb200891-0cae-432f-b1f1-6137e2a1f53b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 17:48:19.286104 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:19.286073    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"
Jul 28 17:48:28.410299 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:28.410265    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"]
Jul 28 17:48:28.410664 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:28.410531    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25" podUID="eb200891-0cae-432f-b1f1-6137e2a1f53b" containerName="kube-rbac-proxy" containerID="cri-o://cfc114537e9194bb2c431678c43c1894019f1988fb08d1bd59aabf495e7e091a" gracePeriod=30
Jul 28 17:48:28.410664 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:28.410517    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25" podUID="eb200891-0cae-432f-b1f1-6137e2a1f53b" containerName="kserve-container" containerID="cri-o://8648d7468eba71765cbb9a8fabe56398b515311cca9a9a405a239d5ac30d08f0" gracePeriod=30
Jul 28 17:48:28.431553 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:28.431530    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:28.717699 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:28.717617    2581 generic.go:356] "Generic (PLEG): container finished" podID="eb200891-0cae-432f-b1f1-6137e2a1f53b" containerID="cfc114537e9194bb2c431678c43c1894019f1988fb08d1bd59aabf495e7e091a" exitCode=2
Jul 28 17:48:28.717699 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:28.717668    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25" event={"ID":"eb200891-0cae-432f-b1f1-6137e2a1f53b","Type":"ContainerDied","Data":"cfc114537e9194bb2c431678c43c1894019f1988fb08d1bd59aabf495e7e091a"}
Jul 28 17:48:29.285898 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:29.285861    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25" podUID="eb200891-0cae-432f-b1f1-6137e2a1f53b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 17:48:30.425122 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:30.425089    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25" podUID="eb200891-0cae-432f-b1f1-6137e2a1f53b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 28 17:48:32.733617 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:32.733532    2581 generic.go:356] "Generic (PLEG): container finished" podID="eb200891-0cae-432f-b1f1-6137e2a1f53b" containerID="8648d7468eba71765cbb9a8fabe56398b515311cca9a9a405a239d5ac30d08f0" exitCode=0
Jul 28 17:48:32.733617 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:32.733571    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25" event={"ID":"eb200891-0cae-432f-b1f1-6137e2a1f53b","Type":"ContainerDied","Data":"8648d7468eba71765cbb9a8fabe56398b515311cca9a9a405a239d5ac30d08f0"}
Jul 28 17:48:33.048156 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.048133    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"
Jul 28 17:48:33.203338 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.203298    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb200891-0cae-432f-b1f1-6137e2a1f53b-kube-rbac-proxy-sar-config\") pod \"eb200891-0cae-432f-b1f1-6137e2a1f53b\" (UID: \"eb200891-0cae-432f-b1f1-6137e2a1f53b\") "
Jul 28 17:48:33.203537 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.203370    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8wbw\" (UniqueName: \"kubernetes.io/projected/eb200891-0cae-432f-b1f1-6137e2a1f53b-kube-api-access-w8wbw\") pod \"eb200891-0cae-432f-b1f1-6137e2a1f53b\" (UID: \"eb200891-0cae-432f-b1f1-6137e2a1f53b\") "
Jul 28 17:48:33.203537 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.203516    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb200891-0cae-432f-b1f1-6137e2a1f53b-proxy-tls\") pod \"eb200891-0cae-432f-b1f1-6137e2a1f53b\" (UID: \"eb200891-0cae-432f-b1f1-6137e2a1f53b\") "
Jul 28 17:48:33.203654 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.203581    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb200891-0cae-432f-b1f1-6137e2a1f53b-kserve-provision-location\") pod \"eb200891-0cae-432f-b1f1-6137e2a1f53b\" (UID: \"eb200891-0cae-432f-b1f1-6137e2a1f53b\") "
Jul 28 17:48:33.203654 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.203622    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb200891-0cae-432f-b1f1-6137e2a1f53b-cabundle-cert\") pod \"eb200891-0cae-432f-b1f1-6137e2a1f53b\" (UID: \"eb200891-0cae-432f-b1f1-6137e2a1f53b\") "
Jul 28 17:48:33.203751 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.203657    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb200891-0cae-432f-b1f1-6137e2a1f53b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eb200891-0cae-432f-b1f1-6137e2a1f53b" (UID: "eb200891-0cae-432f-b1f1-6137e2a1f53b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:48:33.203830 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.203812    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb200891-0cae-432f-b1f1-6137e2a1f53b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:48:33.203997 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.203971    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb200891-0cae-432f-b1f1-6137e2a1f53b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eb200891-0cae-432f-b1f1-6137e2a1f53b" (UID: "eb200891-0cae-432f-b1f1-6137e2a1f53b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:48:33.204074 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.204026    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb200891-0cae-432f-b1f1-6137e2a1f53b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eb200891-0cae-432f-b1f1-6137e2a1f53b" (UID: "eb200891-0cae-432f-b1f1-6137e2a1f53b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:48:33.205331 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.205308    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb200891-0cae-432f-b1f1-6137e2a1f53b-kube-api-access-w8wbw" (OuterVolumeSpecName: "kube-api-access-w8wbw") pod "eb200891-0cae-432f-b1f1-6137e2a1f53b" (UID: "eb200891-0cae-432f-b1f1-6137e2a1f53b"). InnerVolumeSpecName "kube-api-access-w8wbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:48:33.205526 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.205503    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb200891-0cae-432f-b1f1-6137e2a1f53b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eb200891-0cae-432f-b1f1-6137e2a1f53b" (UID: "eb200891-0cae-432f-b1f1-6137e2a1f53b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:48:33.305142 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.305083    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb200891-0cae-432f-b1f1-6137e2a1f53b-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:48:33.305142 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.305103    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb200891-0cae-432f-b1f1-6137e2a1f53b-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:48:33.305142 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.305113    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb200891-0cae-432f-b1f1-6137e2a1f53b-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:48:33.305142 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.305122    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8wbw\" (UniqueName: \"kubernetes.io/projected/eb200891-0cae-432f-b1f1-6137e2a1f53b-kube-api-access-w8wbw\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:48:33.453000 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.452964    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:33.523811 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.523784    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"]
Jul 28 17:48:33.524053 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.524041    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb200891-0cae-432f-b1f1-6137e2a1f53b" containerName="storage-initializer"
Jul 28 17:48:33.524113 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.524055    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb200891-0cae-432f-b1f1-6137e2a1f53b" containerName="storage-initializer"
Jul 28 17:48:33.524113 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.524064    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb200891-0cae-432f-b1f1-6137e2a1f53b" containerName="kube-rbac-proxy"
Jul 28 17:48:33.524113 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.524070    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb200891-0cae-432f-b1f1-6137e2a1f53b" containerName="kube-rbac-proxy"
Jul 28 17:48:33.524113 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.524080    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb200891-0cae-432f-b1f1-6137e2a1f53b" containerName="kserve-container"
Jul 28 17:48:33.524113 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.524086    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb200891-0cae-432f-b1f1-6137e2a1f53b" containerName="kserve-container"
Jul 28 17:48:33.524300 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.524125    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb200891-0cae-432f-b1f1-6137e2a1f53b" containerName="kube-rbac-proxy"
Jul 28 17:48:33.524300 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.524134    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb200891-0cae-432f-b1f1-6137e2a1f53b" containerName="kserve-container"
Jul 28 17:48:33.527136 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.527119    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"
Jul 28 17:48:33.529902 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.529880    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 17:48:33.530030 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.529904    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Jul 28 17:48:33.538479 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.538457    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"]
Jul 28 17:48:33.692996 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.692967    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:48:33.708066 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.708044    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4f5f646-f1fd-4247-adc1-966b2f19dd46-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz\" (UID: \"c4f5f646-f1fd-4247-adc1-966b2f19dd46\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"
Jul 28 17:48:33.708136 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.708075    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpjpw\" (UniqueName: \"kubernetes.io/projected/c4f5f646-f1fd-4247-adc1-966b2f19dd46-kube-api-access-lpjpw\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz\" (UID: \"c4f5f646-f1fd-4247-adc1-966b2f19dd46\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"
Jul 28 17:48:33.708136 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.708104    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4f5f646-f1fd-4247-adc1-966b2f19dd46-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz\" (UID: \"c4f5f646-f1fd-4247-adc1-966b2f19dd46\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"
Jul 28 17:48:33.708205 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.708161    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4f5f646-f1fd-4247-adc1-966b2f19dd46-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz\" (UID: \"c4f5f646-f1fd-4247-adc1-966b2f19dd46\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"
Jul 28 17:48:33.708205 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.708184    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4f5f646-f1fd-4247-adc1-966b2f19dd46-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz\" (UID: \"c4f5f646-f1fd-4247-adc1-966b2f19dd46\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"
Jul 28 17:48:33.738467 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.738437    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25" event={"ID":"eb200891-0cae-432f-b1f1-6137e2a1f53b","Type":"ContainerDied","Data":"733763b52bf39038c76d75cba5a5a2756a29802cce42f882c1d2e864b47f37de"}
Jul 28 17:48:33.738795 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.738469    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"
Jul 28 17:48:33.738795 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.738511    2581 scope.go:117] "RemoveContainer" containerID="cfc114537e9194bb2c431678c43c1894019f1988fb08d1bd59aabf495e7e091a"
Jul 28 17:48:33.746591 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.746574    2581 scope.go:117] "RemoveContainer" containerID="8648d7468eba71765cbb9a8fabe56398b515311cca9a9a405a239d5ac30d08f0"
Jul 28 17:48:33.753308 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.753293    2581 scope.go:117] "RemoveContainer" containerID="3e16beafaf29d05631bd5898e7872c047f45ecf67cd68f266985c4faaff902f9"
Jul 28 17:48:33.762061 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.762040    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"]
Jul 28 17:48:33.766167 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.766146    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-9sc25"]
Jul 28 17:48:33.809165 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.809142    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4f5f646-f1fd-4247-adc1-966b2f19dd46-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz\" (UID: \"c4f5f646-f1fd-4247-adc1-966b2f19dd46\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"
Jul 28 17:48:33.809227 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.809169    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4f5f646-f1fd-4247-adc1-966b2f19dd46-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz\" (UID: \"c4f5f646-f1fd-4247-adc1-966b2f19dd46\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"
Jul 28 17:48:33.809227 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.809209    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4f5f646-f1fd-4247-adc1-966b2f19dd46-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz\" (UID: \"c4f5f646-f1fd-4247-adc1-966b2f19dd46\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"
Jul 28 17:48:33.809302 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.809228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpjpw\" (UniqueName: \"kubernetes.io/projected/c4f5f646-f1fd-4247-adc1-966b2f19dd46-kube-api-access-lpjpw\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz\" (UID: \"c4f5f646-f1fd-4247-adc1-966b2f19dd46\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"
Jul 28 17:48:33.809302 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.809252    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4f5f646-f1fd-4247-adc1-966b2f19dd46-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz\" (UID: \"c4f5f646-f1fd-4247-adc1-966b2f19dd46\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"
Jul 28 17:48:33.809543 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.809524    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4f5f646-f1fd-4247-adc1-966b2f19dd46-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz\" (UID: \"c4f5f646-f1fd-4247-adc1-966b2f19dd46\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"
Jul 28 17:48:33.809933 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.809911    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4f5f646-f1fd-4247-adc1-966b2f19dd46-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz\" (UID: \"c4f5f646-f1fd-4247-adc1-966b2f19dd46\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"
Jul 28 17:48:33.810007 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.809916    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4f5f646-f1fd-4247-adc1-966b2f19dd46-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz\" (UID: \"c4f5f646-f1fd-4247-adc1-966b2f19dd46\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"
Jul 28 17:48:33.811447 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.811426    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4f5f646-f1fd-4247-adc1-966b2f19dd46-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz\" (UID: \"c4f5f646-f1fd-4247-adc1-966b2f19dd46\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"
Jul 28 17:48:33.818550 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.818514    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpjpw\" (UniqueName: \"kubernetes.io/projected/c4f5f646-f1fd-4247-adc1-966b2f19dd46-kube-api-access-lpjpw\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz\" (UID: \"c4f5f646-f1fd-4247-adc1-966b2f19dd46\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"
Jul 28 17:48:33.837375 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.837355    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"
Jul 28 17:48:33.963195 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:33.963049    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"]
Jul 28 17:48:33.965655 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:48:33.965622    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f5f646_f1fd_4247_adc1_966b2f19dd46.slice/crio-0cc898cbfb6c1ffa9aae77e76e86203eb604751f02efb4f255f232ac31763e71 WatchSource:0}: Error finding container 0cc898cbfb6c1ffa9aae77e76e86203eb604751f02efb4f255f232ac31763e71: Status 404 returned error can't find the container with id 0cc898cbfb6c1ffa9aae77e76e86203eb604751f02efb4f255f232ac31763e71
Jul 28 17:48:34.289393 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:34.289318    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb200891-0cae-432f-b1f1-6137e2a1f53b" path="/var/lib/kubelet/pods/eb200891-0cae-432f-b1f1-6137e2a1f53b/volumes"
Jul 28 17:48:34.743868 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:34.743830    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz" event={"ID":"c4f5f646-f1fd-4247-adc1-966b2f19dd46","Type":"ContainerStarted","Data":"a9a7dfbebfb76818a3407aff7515ba2025041af117128bd5842b0687591803bf"}
Jul 28 17:48:34.744258 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:34.743897    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz" event={"ID":"c4f5f646-f1fd-4247-adc1-966b2f19dd46","Type":"ContainerStarted","Data":"0cc898cbfb6c1ffa9aae77e76e86203eb604751f02efb4f255f232ac31763e71"}
Jul 28 17:48:37.754890 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:37.754863    2581 generic.go:356] "Generic (PLEG): container finished" podID="c4f5f646-f1fd-4247-adc1-966b2f19dd46" containerID="a9a7dfbebfb76818a3407aff7515ba2025041af117128bd5842b0687591803bf" exitCode=0
Jul 28 17:48:37.755228 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:48:37.754923    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz" event={"ID":"c4f5f646-f1fd-4247-adc1-966b2f19dd46","Type":"ContainerDied","Data":"a9a7dfbebfb76818a3407aff7515ba2025041af117128bd5842b0687591803bf"}
Jul 28 17:49:12.317722 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:49:12.317697    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 17:49:12.320917 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:49:12.320897    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 17:50:53.296247 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:50:53.296211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz" event={"ID":"c4f5f646-f1fd-4247-adc1-966b2f19dd46","Type":"ContainerStarted","Data":"edc50bc9934dc19a9a8fa0c753fe56bdaac6593541c72141e78c01416f1e579d"}
Jul 28 17:50:53.296867 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:50:53.296283    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz" event={"ID":"c4f5f646-f1fd-4247-adc1-966b2f19dd46","Type":"ContainerStarted","Data":"a8b2ff5725a32fa96ac9a0a2f6c46f2afb96390752f08894eea9d6cdc0c38285"}
Jul 28 17:50:53.296867 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:50:53.296344    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"
Jul 28 17:50:53.327248 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:50:53.327200    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz" podStartSLOduration=2.028047268 podStartE2EDuration="2m20.327182637s" totalImagesPullingTime="2m14.663815949s" totalInitContainerRuntime="3.63531942s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:48:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:50:53.32649794 +0000 UTC m=+2501.601114373" watchObservedRunningTime="2026-07-28 17:50:53.327182637 +0000 UTC m=+2501.601799074"
Jul 28 17:50:54.300204 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:50:54.300174    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"
Jul 28 17:51:00.307954 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:00.307923    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"
Jul 28 17:51:11.950835 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:11.950755    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:30.311743 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:30.311708    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"
Jul 28 17:51:33.718541 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:33.718507    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"]
Jul 28 17:51:33.719011 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:33.718728    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz" podUID="c4f5f646-f1fd-4247-adc1-966b2f19dd46" containerName="kserve-container" containerID="cri-o://a8b2ff5725a32fa96ac9a0a2f6c46f2afb96390752f08894eea9d6cdc0c38285" gracePeriod=30
Jul 28 17:51:33.719011 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:33.718756    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz" podUID="c4f5f646-f1fd-4247-adc1-966b2f19dd46" containerName="kube-rbac-proxy" containerID="cri-o://edc50bc9934dc19a9a8fa0c753fe56bdaac6593541c72141e78c01416f1e579d" gracePeriod=30
Jul 28 17:51:33.753064 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:33.753016    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:34.439429 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:34.439396    2581 generic.go:356] "Generic (PLEG): container finished" podID="c4f5f646-f1fd-4247-adc1-966b2f19dd46" containerID="edc50bc9934dc19a9a8fa0c753fe56bdaac6593541c72141e78c01416f1e579d" exitCode=2
Jul 28 17:51:34.439612 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:34.439484    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz" event={"ID":"c4f5f646-f1fd-4247-adc1-966b2f19dd46","Type":"ContainerDied","Data":"edc50bc9934dc19a9a8fa0c753fe56bdaac6593541c72141e78c01416f1e579d"}
Jul 28 17:51:34.656686 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:34.656663    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"
Jul 28 17:51:34.721555 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:34.721487    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpjpw\" (UniqueName: \"kubernetes.io/projected/c4f5f646-f1fd-4247-adc1-966b2f19dd46-kube-api-access-lpjpw\") pod \"c4f5f646-f1fd-4247-adc1-966b2f19dd46\" (UID: \"c4f5f646-f1fd-4247-adc1-966b2f19dd46\") "
Jul 28 17:51:34.721555 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:34.721544    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4f5f646-f1fd-4247-adc1-966b2f19dd46-kserve-provision-location\") pod \"c4f5f646-f1fd-4247-adc1-966b2f19dd46\" (UID: \"c4f5f646-f1fd-4247-adc1-966b2f19dd46\") "
Jul 28 17:51:34.722034 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:34.721578    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4f5f646-f1fd-4247-adc1-966b2f19dd46-proxy-tls\") pod \"c4f5f646-f1fd-4247-adc1-966b2f19dd46\" (UID: \"c4f5f646-f1fd-4247-adc1-966b2f19dd46\") "
Jul 28 17:51:34.722034 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:34.721602    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4f5f646-f1fd-4247-adc1-966b2f19dd46-cabundle-cert\") pod \"c4f5f646-f1fd-4247-adc1-966b2f19dd46\" (UID: \"c4f5f646-f1fd-4247-adc1-966b2f19dd46\") "
Jul 28 17:51:34.722034 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:34.721670    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4f5f646-f1fd-4247-adc1-966b2f19dd46-kube-rbac-proxy-sar-config\") pod \"c4f5f646-f1fd-4247-adc1-966b2f19dd46\" (UID: \"c4f5f646-f1fd-4247-adc1-966b2f19dd46\") "
Jul 28 17:51:34.722034 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:34.721883    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f5f646-f1fd-4247-adc1-966b2f19dd46-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c4f5f646-f1fd-4247-adc1-966b2f19dd46" (UID: "c4f5f646-f1fd-4247-adc1-966b2f19dd46"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:51:34.722228 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:34.722033    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f5f646-f1fd-4247-adc1-966b2f19dd46-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c4f5f646-f1fd-4247-adc1-966b2f19dd46" (UID: "c4f5f646-f1fd-4247-adc1-966b2f19dd46"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:51:34.722228 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:34.722066    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f5f646-f1fd-4247-adc1-966b2f19dd46-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c4f5f646-f1fd-4247-adc1-966b2f19dd46" (UID: "c4f5f646-f1fd-4247-adc1-966b2f19dd46"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:51:34.723563 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:34.723542    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4f5f646-f1fd-4247-adc1-966b2f19dd46-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c4f5f646-f1fd-4247-adc1-966b2f19dd46" (UID: "c4f5f646-f1fd-4247-adc1-966b2f19dd46"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:51:34.723639 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:34.723622    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f5f646-f1fd-4247-adc1-966b2f19dd46-kube-api-access-lpjpw" (OuterVolumeSpecName: "kube-api-access-lpjpw") pod "c4f5f646-f1fd-4247-adc1-966b2f19dd46" (UID: "c4f5f646-f1fd-4247-adc1-966b2f19dd46"). InnerVolumeSpecName "kube-api-access-lpjpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:51:34.822384 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:34.822356    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpjpw\" (UniqueName: \"kubernetes.io/projected/c4f5f646-f1fd-4247-adc1-966b2f19dd46-kube-api-access-lpjpw\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:51:34.822384 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:34.822380    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c4f5f646-f1fd-4247-adc1-966b2f19dd46-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:51:34.822384 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:34.822392    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c4f5f646-f1fd-4247-adc1-966b2f19dd46-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:51:34.822582 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:34.822400    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c4f5f646-f1fd-4247-adc1-966b2f19dd46-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:51:34.822582 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:34.822409    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c4f5f646-f1fd-4247-adc1-966b2f19dd46-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:51:35.444492 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:35.444452    2581 generic.go:356] "Generic (PLEG): container finished" podID="c4f5f646-f1fd-4247-adc1-966b2f19dd46" containerID="a8b2ff5725a32fa96ac9a0a2f6c46f2afb96390752f08894eea9d6cdc0c38285" exitCode=0
Jul 28 17:51:35.444492 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:35.444492    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz" event={"ID":"c4f5f646-f1fd-4247-adc1-966b2f19dd46","Type":"ContainerDied","Data":"a8b2ff5725a32fa96ac9a0a2f6c46f2afb96390752f08894eea9d6cdc0c38285"}
Jul 28 17:51:35.444718 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:35.444512    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"
Jul 28 17:51:35.444718 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:35.444547    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz" event={"ID":"c4f5f646-f1fd-4247-adc1-966b2f19dd46","Type":"ContainerDied","Data":"0cc898cbfb6c1ffa9aae77e76e86203eb604751f02efb4f255f232ac31763e71"}
Jul 28 17:51:35.444718 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:35.444575    2581 scope.go:117] "RemoveContainer" containerID="edc50bc9934dc19a9a8fa0c753fe56bdaac6593541c72141e78c01416f1e579d"
Jul 28 17:51:35.453262 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:35.453122    2581 scope.go:117] "RemoveContainer" containerID="a8b2ff5725a32fa96ac9a0a2f6c46f2afb96390752f08894eea9d6cdc0c38285"
Jul 28 17:51:35.463032 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:35.461601    2581 scope.go:117] "RemoveContainer" containerID="a9a7dfbebfb76818a3407aff7515ba2025041af117128bd5842b0687591803bf"
Jul 28 17:51:35.469839 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:35.469817    2581 scope.go:117] "RemoveContainer" containerID="edc50bc9934dc19a9a8fa0c753fe56bdaac6593541c72141e78c01416f1e579d"
Jul 28 17:51:35.470049 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:35.470028    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"]
Jul 28 17:51:35.470107 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:51:35.470072    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edc50bc9934dc19a9a8fa0c753fe56bdaac6593541c72141e78c01416f1e579d\": container with ID starting with edc50bc9934dc19a9a8fa0c753fe56bdaac6593541c72141e78c01416f1e579d not found: ID does not exist" containerID="edc50bc9934dc19a9a8fa0c753fe56bdaac6593541c72141e78c01416f1e579d"
Jul 28 17:51:35.470107 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:35.470098    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edc50bc9934dc19a9a8fa0c753fe56bdaac6593541c72141e78c01416f1e579d"} err="failed to get container status \"edc50bc9934dc19a9a8fa0c753fe56bdaac6593541c72141e78c01416f1e579d\": rpc error: code = NotFound desc = could not find container \"edc50bc9934dc19a9a8fa0c753fe56bdaac6593541c72141e78c01416f1e579d\": container with ID starting with edc50bc9934dc19a9a8fa0c753fe56bdaac6593541c72141e78c01416f1e579d not found: ID does not exist"
Jul 28 17:51:35.470199 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:35.470114    2581 scope.go:117] "RemoveContainer" containerID="a8b2ff5725a32fa96ac9a0a2f6c46f2afb96390752f08894eea9d6cdc0c38285"
Jul 28 17:51:35.470366 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:51:35.470350    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8b2ff5725a32fa96ac9a0a2f6c46f2afb96390752f08894eea9d6cdc0c38285\": container with ID starting with a8b2ff5725a32fa96ac9a0a2f6c46f2afb96390752f08894eea9d6cdc0c38285 not found: ID does not exist" containerID="a8b2ff5725a32fa96ac9a0a2f6c46f2afb96390752f08894eea9d6cdc0c38285"
Jul 28 17:51:35.470418 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:35.470368    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8b2ff5725a32fa96ac9a0a2f6c46f2afb96390752f08894eea9d6cdc0c38285"} err="failed to get container status \"a8b2ff5725a32fa96ac9a0a2f6c46f2afb96390752f08894eea9d6cdc0c38285\": rpc error: code = NotFound desc = could not find container \"a8b2ff5725a32fa96ac9a0a2f6c46f2afb96390752f08894eea9d6cdc0c38285\": container with ID starting with a8b2ff5725a32fa96ac9a0a2f6c46f2afb96390752f08894eea9d6cdc0c38285 not found: ID does not exist"
Jul 28 17:51:35.470418 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:35.470380    2581 scope.go:117] "RemoveContainer" containerID="a9a7dfbebfb76818a3407aff7515ba2025041af117128bd5842b0687591803bf"
Jul 28 17:51:35.470608 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:51:35.470594    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9a7dfbebfb76818a3407aff7515ba2025041af117128bd5842b0687591803bf\": container with ID starting with a9a7dfbebfb76818a3407aff7515ba2025041af117128bd5842b0687591803bf not found: ID does not exist" containerID="a9a7dfbebfb76818a3407aff7515ba2025041af117128bd5842b0687591803bf"
Jul 28 17:51:35.470648 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:35.470612    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9a7dfbebfb76818a3407aff7515ba2025041af117128bd5842b0687591803bf"} err="failed to get container status \"a9a7dfbebfb76818a3407aff7515ba2025041af117128bd5842b0687591803bf\": rpc error: code = NotFound desc = could not find container \"a9a7dfbebfb76818a3407aff7515ba2025041af117128bd5842b0687591803bf\": container with ID starting with a9a7dfbebfb76818a3407aff7515ba2025041af117128bd5842b0687591803bf not found: ID does not exist"
Jul 28 17:51:35.474144 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:35.474125    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-jdtfz"]
Jul 28 17:51:36.288951 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:36.288919    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4f5f646-f1fd-4247-adc1-966b2f19dd46" path="/var/lib/kubelet/pods/c4f5f646-f1fd-4247-adc1-966b2f19dd46/volumes"
Jul 28 17:51:38.866355 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:38.866299    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:39.001502 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.001468    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"]
Jul 28 17:51:39.001715 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.001704    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4f5f646-f1fd-4247-adc1-966b2f19dd46" containerName="kserve-container"
Jul 28 17:51:39.001760 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.001716    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4f5f646-f1fd-4247-adc1-966b2f19dd46" containerName="kserve-container"
Jul 28 17:51:39.001760 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.001733    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4f5f646-f1fd-4247-adc1-966b2f19dd46" containerName="storage-initializer"
Jul 28 17:51:39.001760 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.001738    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4f5f646-f1fd-4247-adc1-966b2f19dd46" containerName="storage-initializer"
Jul 28 17:51:39.001760 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.001744    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4f5f646-f1fd-4247-adc1-966b2f19dd46" containerName="kube-rbac-proxy"
Jul 28 17:51:39.001760 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.001750    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4f5f646-f1fd-4247-adc1-966b2f19dd46" containerName="kube-rbac-proxy"
Jul 28 17:51:39.001918 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.001793    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4f5f646-f1fd-4247-adc1-966b2f19dd46" containerName="kube-rbac-proxy"
Jul 28 17:51:39.001918 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.001801    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4f5f646-f1fd-4247-adc1-966b2f19dd46" containerName="kserve-container"
Jul 28 17:51:39.005740 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.005723    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"
Jul 28 17:51:39.011212 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.011187    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 28 17:51:39.011346 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.011220    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:51:39.011346 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.011188    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:51:39.011510 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.011497    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:51:39.012499 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.012481    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 17:51:39.012602 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.012565    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Jul 28 17:51:39.012602 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.012585    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:51:39.023757 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.023729    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"]
Jul 28 17:51:39.049447 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.049425    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:51:39.052569 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.052551    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb\" (UID: \"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"
Jul 28 17:51:39.052629 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.052591    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb\" (UID: \"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"
Jul 28 17:51:39.052629 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.052621    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb\" (UID: \"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"
Jul 28 17:51:39.052704 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.052643    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb\" (UID: \"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"
Jul 28 17:51:39.052704 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.052696    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb872\" (UniqueName: \"kubernetes.io/projected/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-kube-api-access-fb872\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb\" (UID: \"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"
Jul 28 17:51:39.153250 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.153228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb\" (UID: \"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"
Jul 28 17:51:39.153358 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.153259    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb\" (UID: \"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"
Jul 28 17:51:39.153358 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.153289    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb872\" (UniqueName: \"kubernetes.io/projected/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-kube-api-access-fb872\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb\" (UID: \"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"
Jul 28 17:51:39.153358 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.153323    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb\" (UID: \"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"
Jul 28 17:51:39.153478 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.153366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb\" (UID: \"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"
Jul 28 17:51:39.153760 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.153739    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb\" (UID: \"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"
Jul 28 17:51:39.154089 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.154071    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb\" (UID: \"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"
Jul 28 17:51:39.154132 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.154108    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb\" (UID: \"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"
Jul 28 17:51:39.155688 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.155671    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb\" (UID: \"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"
Jul 28 17:51:39.165477 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.165457    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb872\" (UniqueName: \"kubernetes.io/projected/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-kube-api-access-fb872\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb\" (UID: \"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"
Jul 28 17:51:39.315051 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.315024    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"
Jul 28 17:51:39.445097 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.445071    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"]
Jul 28 17:51:39.447512 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:51:39.447483    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cdbab34_0ccf_4fd2_8219_bedfa9c6c4a2.slice/crio-4d26f32e197e8d3d559577f3daff7ab73e09aca8d4e18f21990757b59b221f7a WatchSource:0}: Error finding container 4d26f32e197e8d3d559577f3daff7ab73e09aca8d4e18f21990757b59b221f7a: Status 404 returned error can't find the container with id 4d26f32e197e8d3d559577f3daff7ab73e09aca8d4e18f21990757b59b221f7a
Jul 28 17:51:39.460006 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:39.459978    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb" event={"ID":"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2","Type":"ContainerStarted","Data":"4d26f32e197e8d3d559577f3daff7ab73e09aca8d4e18f21990757b59b221f7a"}
Jul 28 17:51:40.465165 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:40.465129    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb" event={"ID":"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2","Type":"ContainerStarted","Data":"afc13dbf11224991bc50520b3dd13584e0667ccc74f7541f52dc1bf185f2de1c"}
Jul 28 17:51:43.476880 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:43.476800    2581 generic.go:356] "Generic (PLEG): container finished" podID="4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2" containerID="afc13dbf11224991bc50520b3dd13584e0667ccc74f7541f52dc1bf185f2de1c" exitCode=0
Jul 28 17:51:43.476880 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:43.476851    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb" event={"ID":"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2","Type":"ContainerDied","Data":"afc13dbf11224991bc50520b3dd13584e0667ccc74f7541f52dc1bf185f2de1c"}
Jul 28 17:51:44.483145 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:44.483108    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb" event={"ID":"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2","Type":"ContainerStarted","Data":"932d82decf325d24c23fce0801cf09bb0cb21a78e701aefa68facb0050b72226"}
Jul 28 17:51:44.483576 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:44.483173    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb" event={"ID":"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2","Type":"ContainerStarted","Data":"c6e96233e8f47e9e36ba47ab936c3f7b90281f3faabb903c2e61abbfa6c87e03"}
Jul 28 17:51:44.483576 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:44.483443    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"
Jul 28 17:51:44.483681 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:44.483585    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"
Jul 28 17:51:44.484784 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:44.484761    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb" podUID="4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 28 17:51:44.506567 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:44.506528    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb" podStartSLOduration=2.787494332 podStartE2EDuration="6.506492179s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.718997847s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:51:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:51:44.506088387 +0000 UTC m=+2552.780704820" watchObservedRunningTime="2026-07-28 17:51:44.506492179 +0000 UTC m=+2552.781108610"
Jul 28 17:51:45.487320 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:45.487274    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb" podUID="4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 28 17:51:50.491833 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:50.491801    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"
Jul 28 17:51:50.492436 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:51:50.492407    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb" podUID="4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 28 17:52:00.493105 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:00.493075    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"
Jul 28 17:52:08.934144 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:08.934111    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"]
Jul 28 17:52:08.934708 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:08.934413    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb" podUID="4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2" containerName="kserve-container" containerID="cri-o://c6e96233e8f47e9e36ba47ab936c3f7b90281f3faabb903c2e61abbfa6c87e03" gracePeriod=30
Jul 28 17:52:08.934708 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:08.934439    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb" podUID="4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2" containerName="kube-rbac-proxy" containerID="cri-o://932d82decf325d24c23fce0801cf09bb0cb21a78e701aefa68facb0050b72226" gracePeriod=30
Jul 28 17:52:08.953417 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:08.953392    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:52:09.575978 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.575935    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"
Jul 28 17:52:09.577113 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.577089    2581 generic.go:356] "Generic (PLEG): container finished" podID="4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2" containerID="932d82decf325d24c23fce0801cf09bb0cb21a78e701aefa68facb0050b72226" exitCode=2
Jul 28 17:52:09.577113 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.577111    2581 generic.go:356] "Generic (PLEG): container finished" podID="4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2" containerID="c6e96233e8f47e9e36ba47ab936c3f7b90281f3faabb903c2e61abbfa6c87e03" exitCode=0
Jul 28 17:52:09.577242 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.577154    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb" event={"ID":"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2","Type":"ContainerDied","Data":"932d82decf325d24c23fce0801cf09bb0cb21a78e701aefa68facb0050b72226"}
Jul 28 17:52:09.577242 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.577210    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb" event={"ID":"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2","Type":"ContainerDied","Data":"c6e96233e8f47e9e36ba47ab936c3f7b90281f3faabb903c2e61abbfa6c87e03"}
Jul 28 17:52:09.577242 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.577231    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb" event={"ID":"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2","Type":"ContainerDied","Data":"4d26f32e197e8d3d559577f3daff7ab73e09aca8d4e18f21990757b59b221f7a"}
Jul 28 17:52:09.577360 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.577255    2581 scope.go:117] "RemoveContainer" containerID="932d82decf325d24c23fce0801cf09bb0cb21a78e701aefa68facb0050b72226"
Jul 28 17:52:09.584427 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.584410    2581 scope.go:117] "RemoveContainer" containerID="c6e96233e8f47e9e36ba47ab936c3f7b90281f3faabb903c2e61abbfa6c87e03"
Jul 28 17:52:09.591198 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.591182    2581 scope.go:117] "RemoveContainer" containerID="afc13dbf11224991bc50520b3dd13584e0667ccc74f7541f52dc1bf185f2de1c"
Jul 28 17:52:09.598188 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.598168    2581 scope.go:117] "RemoveContainer" containerID="932d82decf325d24c23fce0801cf09bb0cb21a78e701aefa68facb0050b72226"
Jul 28 17:52:09.598442 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:52:09.598420    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"932d82decf325d24c23fce0801cf09bb0cb21a78e701aefa68facb0050b72226\": container with ID starting with 932d82decf325d24c23fce0801cf09bb0cb21a78e701aefa68facb0050b72226 not found: ID does not exist" containerID="932d82decf325d24c23fce0801cf09bb0cb21a78e701aefa68facb0050b72226"
Jul 28 17:52:09.598503 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.598450    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"932d82decf325d24c23fce0801cf09bb0cb21a78e701aefa68facb0050b72226"} err="failed to get container status \"932d82decf325d24c23fce0801cf09bb0cb21a78e701aefa68facb0050b72226\": rpc error: code = NotFound desc = could not find container \"932d82decf325d24c23fce0801cf09bb0cb21a78e701aefa68facb0050b72226\": container with ID starting with 932d82decf325d24c23fce0801cf09bb0cb21a78e701aefa68facb0050b72226 not found: ID does not exist"
Jul 28 17:52:09.598503 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.598468    2581 scope.go:117] "RemoveContainer" containerID="c6e96233e8f47e9e36ba47ab936c3f7b90281f3faabb903c2e61abbfa6c87e03"
Jul 28 17:52:09.598713 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:52:09.598697    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6e96233e8f47e9e36ba47ab936c3f7b90281f3faabb903c2e61abbfa6c87e03\": container with ID starting with c6e96233e8f47e9e36ba47ab936c3f7b90281f3faabb903c2e61abbfa6c87e03 not found: ID does not exist" containerID="c6e96233e8f47e9e36ba47ab936c3f7b90281f3faabb903c2e61abbfa6c87e03"
Jul 28 17:52:09.598759 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.598719    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e96233e8f47e9e36ba47ab936c3f7b90281f3faabb903c2e61abbfa6c87e03"} err="failed to get container status \"c6e96233e8f47e9e36ba47ab936c3f7b90281f3faabb903c2e61abbfa6c87e03\": rpc error: code = NotFound desc = could not find container \"c6e96233e8f47e9e36ba47ab936c3f7b90281f3faabb903c2e61abbfa6c87e03\": container with ID starting with c6e96233e8f47e9e36ba47ab936c3f7b90281f3faabb903c2e61abbfa6c87e03 not found: ID does not exist"
Jul 28 17:52:09.598759 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.598736    2581 scope.go:117] "RemoveContainer" containerID="afc13dbf11224991bc50520b3dd13584e0667ccc74f7541f52dc1bf185f2de1c"
Jul 28 17:52:09.599001 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:52:09.598982    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afc13dbf11224991bc50520b3dd13584e0667ccc74f7541f52dc1bf185f2de1c\": container with ID starting with afc13dbf11224991bc50520b3dd13584e0667ccc74f7541f52dc1bf185f2de1c not found: ID does not exist" containerID="afc13dbf11224991bc50520b3dd13584e0667ccc74f7541f52dc1bf185f2de1c"
Jul 28 17:52:09.599066 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.599007    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc13dbf11224991bc50520b3dd13584e0667ccc74f7541f52dc1bf185f2de1c"} err="failed to get container status \"afc13dbf11224991bc50520b3dd13584e0667ccc74f7541f52dc1bf185f2de1c\": rpc error: code = NotFound desc = could not find container \"afc13dbf11224991bc50520b3dd13584e0667ccc74f7541f52dc1bf185f2de1c\": container with ID starting with afc13dbf11224991bc50520b3dd13584e0667ccc74f7541f52dc1bf185f2de1c not found: ID does not exist"
Jul 28 17:52:09.599066 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.599023    2581 scope.go:117] "RemoveContainer" containerID="932d82decf325d24c23fce0801cf09bb0cb21a78e701aefa68facb0050b72226"
Jul 28 17:52:09.599248 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.599232    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"932d82decf325d24c23fce0801cf09bb0cb21a78e701aefa68facb0050b72226"} err="failed to get container status \"932d82decf325d24c23fce0801cf09bb0cb21a78e701aefa68facb0050b72226\": rpc error: code = NotFound desc = could not find container \"932d82decf325d24c23fce0801cf09bb0cb21a78e701aefa68facb0050b72226\": container with ID starting with 932d82decf325d24c23fce0801cf09bb0cb21a78e701aefa68facb0050b72226 not found: ID does not exist"
Jul 28 17:52:09.599287 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.599248    2581 scope.go:117] "RemoveContainer" containerID="c6e96233e8f47e9e36ba47ab936c3f7b90281f3faabb903c2e61abbfa6c87e03"
Jul 28 17:52:09.599596 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.599535    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e96233e8f47e9e36ba47ab936c3f7b90281f3faabb903c2e61abbfa6c87e03"} err="failed to get container status \"c6e96233e8f47e9e36ba47ab936c3f7b90281f3faabb903c2e61abbfa6c87e03\": rpc error: code = NotFound desc = could not find container \"c6e96233e8f47e9e36ba47ab936c3f7b90281f3faabb903c2e61abbfa6c87e03\": container with ID starting with c6e96233e8f47e9e36ba47ab936c3f7b90281f3faabb903c2e61abbfa6c87e03 not found: ID does not exist"
Jul 28 17:52:09.599713 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.599597    2581 scope.go:117] "RemoveContainer" containerID="afc13dbf11224991bc50520b3dd13584e0667ccc74f7541f52dc1bf185f2de1c"
Jul 28 17:52:09.599922 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.599898    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc13dbf11224991bc50520b3dd13584e0667ccc74f7541f52dc1bf185f2de1c"} err="failed to get container status \"afc13dbf11224991bc50520b3dd13584e0667ccc74f7541f52dc1bf185f2de1c\": rpc error: code = NotFound desc = could not find container \"afc13dbf11224991bc50520b3dd13584e0667ccc74f7541f52dc1bf185f2de1c\": container with ID starting with afc13dbf11224991bc50520b3dd13584e0667ccc74f7541f52dc1bf185f2de1c not found: ID does not exist"
Jul 28 17:52:09.654498 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.654477    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-kube-rbac-proxy-sar-config\") pod \"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2\" (UID: \"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2\") "
Jul 28 17:52:09.654616 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.654509    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-proxy-tls\") pod \"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2\" (UID: \"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2\") "
Jul 28 17:52:09.654616 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.654533    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-cabundle-cert\") pod \"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2\" (UID: \"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2\") "
Jul 28 17:52:09.654616 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.654553    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fb872\" (UniqueName: \"kubernetes.io/projected/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-kube-api-access-fb872\") pod \"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2\" (UID: \"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2\") "
Jul 28 17:52:09.654743 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.654679    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-kserve-provision-location\") pod \"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2\" (UID: \"4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2\") "
Jul 28 17:52:09.654997 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.654965    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2" (UID: "4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:52:09.655079 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.654964    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2" (UID: "4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:52:09.655119 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.655075    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2" (UID: "4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:52:09.656492 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.656473    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2" (UID: "4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:52:09.656564 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.656525    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-kube-api-access-fb872" (OuterVolumeSpecName: "kube-api-access-fb872") pod "4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2" (UID: "4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2"). InnerVolumeSpecName "kube-api-access-fb872". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:52:09.756245 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.756164    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:52:09.756245 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.756200    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:52:09.756245 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.756213    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb872\" (UniqueName: \"kubernetes.io/projected/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-kube-api-access-fb872\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:52:09.756245 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.756228    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:52:09.756245 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:09.756240    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:52:10.581768 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:10.581691    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"
Jul 28 17:52:10.604249 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:10.604223    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"]
Jul 28 17:52:10.609690 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:10.609667    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-lkfgb"]
Jul 28 17:52:12.289989 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:12.289933    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2" path="/var/lib/kubelet/pods/4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2/volumes"
Jul 28 17:52:14.127320 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.127267    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:52:14.181851 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.181813    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"]
Jul 28 17:52:14.182140 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.182123    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2" containerName="kserve-container"
Jul 28 17:52:14.182226 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.182142    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2" containerName="kserve-container"
Jul 28 17:52:14.182226 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.182165    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2" containerName="kube-rbac-proxy"
Jul 28 17:52:14.182226 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.182172    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2" containerName="kube-rbac-proxy"
Jul 28 17:52:14.182226 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.182188    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2" containerName="storage-initializer"
Jul 28 17:52:14.182226 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.182196    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2" containerName="storage-initializer"
Jul 28 17:52:14.182493 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.182272    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2" containerName="kube-rbac-proxy"
Jul 28 17:52:14.182493 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.182283    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cdbab34-0ccf-4fd2-8219-bedfa9c6c4a2" containerName="kserve-container"
Jul 28 17:52:14.186699 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.186677    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"
Jul 28 17:52:14.190103 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.190076    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Jul 28 17:52:14.190227 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.190157    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 17:52:14.190227 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.190168    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:52:14.191719 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.191702    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 17:52:14.191805 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.191745    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:52:14.191994 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.191974    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:52:14.192063 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.192041    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:52:14.199587 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.199564    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"]
Jul 28 17:52:14.289871 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.289842    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p\" (UID: \"9741a2a3-8c17-4880-9b14-5fd009e0fa1a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"
Jul 28 17:52:14.290102 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.289880    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p\" (UID: \"9741a2a3-8c17-4880-9b14-5fd009e0fa1a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"
Jul 28 17:52:14.290102 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.289900    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p\" (UID: \"9741a2a3-8c17-4880-9b14-5fd009e0fa1a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"
Jul 28 17:52:14.290102 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.289925    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q57w\" (UniqueName: \"kubernetes.io/projected/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-kube-api-access-4q57w\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p\" (UID: \"9741a2a3-8c17-4880-9b14-5fd009e0fa1a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"
Jul 28 17:52:14.290102 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.290019    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p\" (UID: \"9741a2a3-8c17-4880-9b14-5fd009e0fa1a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"
Jul 28 17:52:14.390826 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.390744    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p\" (UID: \"9741a2a3-8c17-4880-9b14-5fd009e0fa1a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"
Jul 28 17:52:14.390826 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.390787    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p\" (UID: \"9741a2a3-8c17-4880-9b14-5fd009e0fa1a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"
Jul 28 17:52:14.390826 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.390809    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p\" (UID: \"9741a2a3-8c17-4880-9b14-5fd009e0fa1a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"
Jul 28 17:52:14.391103 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.390837    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q57w\" (UniqueName: \"kubernetes.io/projected/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-kube-api-access-4q57w\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p\" (UID: \"9741a2a3-8c17-4880-9b14-5fd009e0fa1a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"
Jul 28 17:52:14.391103 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.390876    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p\" (UID: \"9741a2a3-8c17-4880-9b14-5fd009e0fa1a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"
Jul 28 17:52:14.391103 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:52:14.390915    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-serving-cert: secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Jul 28 17:52:14.391103 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:52:14.391010    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-proxy-tls podName:9741a2a3-8c17-4880-9b14-5fd009e0fa1a nodeName:}" failed. No retries permitted until 2026-07-28 17:52:14.890989387 +0000 UTC m=+2583.165605801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-proxy-tls") pod "isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p" (UID: "9741a2a3-8c17-4880-9b14-5fd009e0fa1a") : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Jul 28 17:52:14.391334 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.391311    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p\" (UID: \"9741a2a3-8c17-4880-9b14-5fd009e0fa1a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"
Jul 28 17:52:14.391529 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.391510    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p\" (UID: \"9741a2a3-8c17-4880-9b14-5fd009e0fa1a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"
Jul 28 17:52:14.391586 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.391520    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p\" (UID: \"9741a2a3-8c17-4880-9b14-5fd009e0fa1a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"
Jul 28 17:52:14.402334 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.402316    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q57w\" (UniqueName: \"kubernetes.io/projected/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-kube-api-access-4q57w\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p\" (UID: \"9741a2a3-8c17-4880-9b14-5fd009e0fa1a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"
Jul 28 17:52:14.521149 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.521120    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:52:14.893812 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.893780    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p\" (UID: \"9741a2a3-8c17-4880-9b14-5fd009e0fa1a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"
Jul 28 17:52:14.896101 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:14.896072    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p\" (UID: \"9741a2a3-8c17-4880-9b14-5fd009e0fa1a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"
Jul 28 17:52:15.096957 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:15.096911    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"
Jul 28 17:52:15.239383 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:15.239358    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"]
Jul 28 17:52:15.241195 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:52:15.241168    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9741a2a3_8c17_4880_9b14_5fd009e0fa1a.slice/crio-a1976530451b4ce83c56c37cf6efe1e12a2486cfbef43e65bebc285a9a08803b WatchSource:0}: Error finding container a1976530451b4ce83c56c37cf6efe1e12a2486cfbef43e65bebc285a9a08803b: Status 404 returned error can't find the container with id a1976530451b4ce83c56c37cf6efe1e12a2486cfbef43e65bebc285a9a08803b
Jul 28 17:52:15.598660 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:15.598627    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p" event={"ID":"9741a2a3-8c17-4880-9b14-5fd009e0fa1a","Type":"ContainerStarted","Data":"8f4b744a8db58636b9dace130427a8f9ba6b92fe8c1d16566cb607c67d9bb179"}
Jul 28 17:52:15.598822 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:15.598681    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p" event={"ID":"9741a2a3-8c17-4880-9b14-5fd009e0fa1a","Type":"ContainerStarted","Data":"a1976530451b4ce83c56c37cf6efe1e12a2486cfbef43e65bebc285a9a08803b"}
Jul 28 17:52:19.613682 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:19.613594    2581 generic.go:356] "Generic (PLEG): container finished" podID="9741a2a3-8c17-4880-9b14-5fd009e0fa1a" containerID="8f4b744a8db58636b9dace130427a8f9ba6b92fe8c1d16566cb607c67d9bb179" exitCode=0
Jul 28 17:52:19.613682 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:19.613668    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p" event={"ID":"9741a2a3-8c17-4880-9b14-5fd009e0fa1a","Type":"ContainerDied","Data":"8f4b744a8db58636b9dace130427a8f9ba6b92fe8c1d16566cb607c67d9bb179"}
Jul 28 17:52:20.619128 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:20.619092    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p" event={"ID":"9741a2a3-8c17-4880-9b14-5fd009e0fa1a","Type":"ContainerStarted","Data":"d25650a06c67b0d8f227b29035eb5247316fe9cab9f3d396a56efd505e614b84"}
Jul 28 17:52:20.619509 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:20.619155    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p" event={"ID":"9741a2a3-8c17-4880-9b14-5fd009e0fa1a","Type":"ContainerStarted","Data":"9ff383977f52ca2607c747fa4f4865f95c34e59d2f66f1ed3420ccc53dec3076"}
Jul 28 17:52:20.619509 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:20.619347    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"
Jul 28 17:52:20.642114 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:20.642071    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p" podStartSLOduration=2.591523334 podStartE2EDuration="6.642055594s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.05053226s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:52:20.641286326 +0000 UTC m=+2588.915902758" watchObservedRunningTime="2026-07-28 17:52:20.642055594 +0000 UTC m=+2588.916672026"
Jul 28 17:52:21.622535 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:21.622510    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"
Jul 28 17:52:27.630992 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:27.630962    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"
Jul 28 17:52:57.634692 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:52:57.634659    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"
Jul 28 17:53:04.228243 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:04.228208    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"]
Jul 28 17:53:04.228622 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:04.228466    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p" podUID="9741a2a3-8c17-4880-9b14-5fd009e0fa1a" containerName="kserve-container" containerID="cri-o://9ff383977f52ca2607c747fa4f4865f95c34e59d2f66f1ed3420ccc53dec3076" gracePeriod=30
Jul 28 17:53:04.228622 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:04.228496    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p" podUID="9741a2a3-8c17-4880-9b14-5fd009e0fa1a" containerName="kube-rbac-proxy" containerID="cri-o://d25650a06c67b0d8f227b29035eb5247316fe9cab9f3d396a56efd505e614b84" gracePeriod=30
Jul 28 17:53:04.266513 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:04.266481    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:04.772666 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:04.772636    2581 generic.go:356] "Generic (PLEG): container finished" podID="9741a2a3-8c17-4880-9b14-5fd009e0fa1a" containerID="d25650a06c67b0d8f227b29035eb5247316fe9cab9f3d396a56efd505e614b84" exitCode=2
Jul 28 17:53:04.772835 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:04.772671    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p" event={"ID":"9741a2a3-8c17-4880-9b14-5fd009e0fa1a","Type":"ContainerDied","Data":"d25650a06c67b0d8f227b29035eb5247316fe9cab9f3d396a56efd505e614b84"}
Jul 28 17:53:05.364825 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.364800    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"
Jul 28 17:53:05.442558 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.442532    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-proxy-tls\") pod \"9741a2a3-8c17-4880-9b14-5fd009e0fa1a\" (UID: \"9741a2a3-8c17-4880-9b14-5fd009e0fa1a\") "
Jul 28 17:53:05.442674 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.442623    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4q57w\" (UniqueName: \"kubernetes.io/projected/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-kube-api-access-4q57w\") pod \"9741a2a3-8c17-4880-9b14-5fd009e0fa1a\" (UID: \"9741a2a3-8c17-4880-9b14-5fd009e0fa1a\") "
Jul 28 17:53:05.442674 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.442658    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-kserve-provision-location\") pod \"9741a2a3-8c17-4880-9b14-5fd009e0fa1a\" (UID: \"9741a2a3-8c17-4880-9b14-5fd009e0fa1a\") "
Jul 28 17:53:05.442766 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.442697    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-cabundle-cert\") pod \"9741a2a3-8c17-4880-9b14-5fd009e0fa1a\" (UID: \"9741a2a3-8c17-4880-9b14-5fd009e0fa1a\") "
Jul 28 17:53:05.442766 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.442719    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-kube-rbac-proxy-sar-config\") pod \"9741a2a3-8c17-4880-9b14-5fd009e0fa1a\" (UID: \"9741a2a3-8c17-4880-9b14-5fd009e0fa1a\") "
Jul 28 17:53:05.443103 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.443074    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9741a2a3-8c17-4880-9b14-5fd009e0fa1a" (UID: "9741a2a3-8c17-4880-9b14-5fd009e0fa1a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:53:05.443231 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.443134    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9741a2a3-8c17-4880-9b14-5fd009e0fa1a" (UID: "9741a2a3-8c17-4880-9b14-5fd009e0fa1a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:53:05.443231 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.443138    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9741a2a3-8c17-4880-9b14-5fd009e0fa1a" (UID: "9741a2a3-8c17-4880-9b14-5fd009e0fa1a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:53:05.444590 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.444568    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-kube-api-access-4q57w" (OuterVolumeSpecName: "kube-api-access-4q57w") pod "9741a2a3-8c17-4880-9b14-5fd009e0fa1a" (UID: "9741a2a3-8c17-4880-9b14-5fd009e0fa1a"). InnerVolumeSpecName "kube-api-access-4q57w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:53:05.444678 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.444623    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9741a2a3-8c17-4880-9b14-5fd009e0fa1a" (UID: "9741a2a3-8c17-4880-9b14-5fd009e0fa1a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:53:05.543506 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.543484    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:53:05.543506 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.543504    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:53:05.543651 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.543514    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:53:05.543651 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.543522    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q57w\" (UniqueName: \"kubernetes.io/projected/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-kube-api-access-4q57w\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:53:05.543651 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.543532    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9741a2a3-8c17-4880-9b14-5fd009e0fa1a-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:53:05.778295 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.778217    2581 generic.go:356] "Generic (PLEG): container finished" podID="9741a2a3-8c17-4880-9b14-5fd009e0fa1a" containerID="9ff383977f52ca2607c747fa4f4865f95c34e59d2f66f1ed3420ccc53dec3076" exitCode=0
Jul 28 17:53:05.778295 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.778251    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p" event={"ID":"9741a2a3-8c17-4880-9b14-5fd009e0fa1a","Type":"ContainerDied","Data":"9ff383977f52ca2607c747fa4f4865f95c34e59d2f66f1ed3420ccc53dec3076"}
Jul 28 17:53:05.778295 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.778273    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"
Jul 28 17:53:05.778519 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.778301    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p" event={"ID":"9741a2a3-8c17-4880-9b14-5fd009e0fa1a","Type":"ContainerDied","Data":"a1976530451b4ce83c56c37cf6efe1e12a2486cfbef43e65bebc285a9a08803b"}
Jul 28 17:53:05.778519 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.778326    2581 scope.go:117] "RemoveContainer" containerID="d25650a06c67b0d8f227b29035eb5247316fe9cab9f3d396a56efd505e614b84"
Jul 28 17:53:05.786190 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.786171    2581 scope.go:117] "RemoveContainer" containerID="9ff383977f52ca2607c747fa4f4865f95c34e59d2f66f1ed3420ccc53dec3076"
Jul 28 17:53:05.793204 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.793188    2581 scope.go:117] "RemoveContainer" containerID="8f4b744a8db58636b9dace130427a8f9ba6b92fe8c1d16566cb607c67d9bb179"
Jul 28 17:53:05.799673 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.799652    2581 scope.go:117] "RemoveContainer" containerID="d25650a06c67b0d8f227b29035eb5247316fe9cab9f3d396a56efd505e614b84"
Jul 28 17:53:05.800010 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:53:05.799988    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d25650a06c67b0d8f227b29035eb5247316fe9cab9f3d396a56efd505e614b84\": container with ID starting with d25650a06c67b0d8f227b29035eb5247316fe9cab9f3d396a56efd505e614b84 not found: ID does not exist" containerID="d25650a06c67b0d8f227b29035eb5247316fe9cab9f3d396a56efd505e614b84"
Jul 28 17:53:05.800100 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.800018    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d25650a06c67b0d8f227b29035eb5247316fe9cab9f3d396a56efd505e614b84"} err="failed to get container status \"d25650a06c67b0d8f227b29035eb5247316fe9cab9f3d396a56efd505e614b84\": rpc error: code = NotFound desc = could not find container \"d25650a06c67b0d8f227b29035eb5247316fe9cab9f3d396a56efd505e614b84\": container with ID starting with d25650a06c67b0d8f227b29035eb5247316fe9cab9f3d396a56efd505e614b84 not found: ID does not exist"
Jul 28 17:53:05.800100 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.800036    2581 scope.go:117] "RemoveContainer" containerID="9ff383977f52ca2607c747fa4f4865f95c34e59d2f66f1ed3420ccc53dec3076"
Jul 28 17:53:05.800411 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:53:05.800390    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ff383977f52ca2607c747fa4f4865f95c34e59d2f66f1ed3420ccc53dec3076\": container with ID starting with 9ff383977f52ca2607c747fa4f4865f95c34e59d2f66f1ed3420ccc53dec3076 not found: ID does not exist" containerID="9ff383977f52ca2607c747fa4f4865f95c34e59d2f66f1ed3420ccc53dec3076"
Jul 28 17:53:05.800469 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.800417    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ff383977f52ca2607c747fa4f4865f95c34e59d2f66f1ed3420ccc53dec3076"} err="failed to get container status \"9ff383977f52ca2607c747fa4f4865f95c34e59d2f66f1ed3420ccc53dec3076\": rpc error: code = NotFound desc = could not find container \"9ff383977f52ca2607c747fa4f4865f95c34e59d2f66f1ed3420ccc53dec3076\": container with ID starting with 9ff383977f52ca2607c747fa4f4865f95c34e59d2f66f1ed3420ccc53dec3076 not found: ID does not exist"
Jul 28 17:53:05.800469 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.800433    2581 scope.go:117] "RemoveContainer" containerID="8f4b744a8db58636b9dace130427a8f9ba6b92fe8c1d16566cb607c67d9bb179"
Jul 28 17:53:05.800675 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:53:05.800659    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f4b744a8db58636b9dace130427a8f9ba6b92fe8c1d16566cb607c67d9bb179\": container with ID starting with 8f4b744a8db58636b9dace130427a8f9ba6b92fe8c1d16566cb607c67d9bb179 not found: ID does not exist" containerID="8f4b744a8db58636b9dace130427a8f9ba6b92fe8c1d16566cb607c67d9bb179"
Jul 28 17:53:05.800721 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.800680    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f4b744a8db58636b9dace130427a8f9ba6b92fe8c1d16566cb607c67d9bb179"} err="failed to get container status \"8f4b744a8db58636b9dace130427a8f9ba6b92fe8c1d16566cb607c67d9bb179\": rpc error: code = NotFound desc = could not find container \"8f4b744a8db58636b9dace130427a8f9ba6b92fe8c1d16566cb607c67d9bb179\": container with ID starting with 8f4b744a8db58636b9dace130427a8f9ba6b92fe8c1d16566cb607c67d9bb179 not found: ID does not exist"
Jul 28 17:53:05.801139 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.801100    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"]
Jul 28 17:53:05.806012 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:05.805986    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-89z7p"]
Jul 28 17:53:06.289614 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:06.289582    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9741a2a3-8c17-4880-9b14-5fd009e0fa1a" path="/var/lib/kubelet/pods/9741a2a3-8c17-4880-9b14-5fd009e0fa1a/volumes"
Jul 28 17:53:09.345951 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.345898    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:09.379165 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.379134    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"]
Jul 28 17:53:09.379378 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.379366    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9741a2a3-8c17-4880-9b14-5fd009e0fa1a" containerName="kube-rbac-proxy"
Jul 28 17:53:09.379420 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.379379    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9741a2a3-8c17-4880-9b14-5fd009e0fa1a" containerName="kube-rbac-proxy"
Jul 28 17:53:09.379420 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.379391    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9741a2a3-8c17-4880-9b14-5fd009e0fa1a" containerName="storage-initializer"
Jul 28 17:53:09.379420 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.379397    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9741a2a3-8c17-4880-9b14-5fd009e0fa1a" containerName="storage-initializer"
Jul 28 17:53:09.379420 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.379406    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9741a2a3-8c17-4880-9b14-5fd009e0fa1a" containerName="kserve-container"
Jul 28 17:53:09.379420 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.379411    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9741a2a3-8c17-4880-9b14-5fd009e0fa1a" containerName="kserve-container"
Jul 28 17:53:09.379571 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.379454    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9741a2a3-8c17-4880-9b14-5fd009e0fa1a" containerName="kube-rbac-proxy"
Jul 28 17:53:09.379571 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.379461    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9741a2a3-8c17-4880-9b14-5fd009e0fa1a" containerName="kserve-container"
Jul 28 17:53:09.384044 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.384028    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"
Jul 28 17:53:09.387448 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.387410    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:53:09.387448 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.387418    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:53:09.387637 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.387577    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Jul 28 17:53:09.387760 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.387743    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:53:09.388132 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.388118    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Jul 28 17:53:09.388704 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.388688    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:53:09.389454 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.389438    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 17:53:09.397364 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.397343    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"]
Jul 28 17:53:09.467699 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.467675    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb\" (UID: \"b33f6041-ae8b-41ef-b608-356bfdbb1e0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"
Jul 28 17:53:09.467841 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.467704    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw5hd\" (UniqueName: \"kubernetes.io/projected/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-kube-api-access-kw5hd\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb\" (UID: \"b33f6041-ae8b-41ef-b608-356bfdbb1e0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"
Jul 28 17:53:09.467841 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.467727    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb\" (UID: \"b33f6041-ae8b-41ef-b608-356bfdbb1e0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"
Jul 28 17:53:09.467841 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.467799    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb\" (UID: \"b33f6041-ae8b-41ef-b608-356bfdbb1e0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"
Jul 28 17:53:09.467841 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.467814    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb\" (UID: \"b33f6041-ae8b-41ef-b608-356bfdbb1e0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"
Jul 28 17:53:09.568377 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.568347    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb\" (UID: \"b33f6041-ae8b-41ef-b608-356bfdbb1e0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"
Jul 28 17:53:09.568516 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.568384    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw5hd\" (UniqueName: \"kubernetes.io/projected/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-kube-api-access-kw5hd\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb\" (UID: \"b33f6041-ae8b-41ef-b608-356bfdbb1e0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"
Jul 28 17:53:09.568516 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.568408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb\" (UID: \"b33f6041-ae8b-41ef-b608-356bfdbb1e0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"
Jul 28 17:53:09.568516 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.568454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb\" (UID: \"b33f6041-ae8b-41ef-b608-356bfdbb1e0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"
Jul 28 17:53:09.568516 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.568471    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb\" (UID: \"b33f6041-ae8b-41ef-b608-356bfdbb1e0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"
Jul 28 17:53:09.568722 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:53:09.568568    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-serving-cert: secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Jul 28 17:53:09.568722 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:53:09.568636    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-proxy-tls podName:b33f6041-ae8b-41ef-b608-356bfdbb1e0c nodeName:}" failed. No retries permitted until 2026-07-28 17:53:10.068613668 +0000 UTC m=+2638.343230083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-proxy-tls") pod "isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb" (UID: "b33f6041-ae8b-41ef-b608-356bfdbb1e0c") : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Jul 28 17:53:09.568865 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.568847    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb\" (UID: \"b33f6041-ae8b-41ef-b608-356bfdbb1e0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"
Jul 28 17:53:09.569075 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.569059    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb\" (UID: \"b33f6041-ae8b-41ef-b608-356bfdbb1e0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"
Jul 28 17:53:09.569142 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.569064    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb\" (UID: \"b33f6041-ae8b-41ef-b608-356bfdbb1e0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"
Jul 28 17:53:09.582474 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.582451    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw5hd\" (UniqueName: \"kubernetes.io/projected/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-kube-api-access-kw5hd\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb\" (UID: \"b33f6041-ae8b-41ef-b608-356bfdbb1e0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"
Jul 28 17:53:09.934081 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:09.934037    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:53:10.071236 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:10.071199    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb\" (UID: \"b33f6041-ae8b-41ef-b608-356bfdbb1e0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"
Jul 28 17:53:10.073552 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:10.073527    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb\" (UID: \"b33f6041-ae8b-41ef-b608-356bfdbb1e0c\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"
Jul 28 17:53:10.293881 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:10.293808    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"
Jul 28 17:53:10.419831 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:10.419800    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"]
Jul 28 17:53:10.422891 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:53:10.422864    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33f6041_ae8b_41ef_b608_356bfdbb1e0c.slice/crio-69a0d60b94df39809221fd1d261b13953428524136a4b0cfd09e4bf5f2ffd635 WatchSource:0}: Error finding container 69a0d60b94df39809221fd1d261b13953428524136a4b0cfd09e4bf5f2ffd635: Status 404 returned error can't find the container with id 69a0d60b94df39809221fd1d261b13953428524136a4b0cfd09e4bf5f2ffd635
Jul 28 17:53:10.797876 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:10.797834    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb" event={"ID":"b33f6041-ae8b-41ef-b608-356bfdbb1e0c","Type":"ContainerStarted","Data":"240f5b645ff0e974fdefdb35892ceac43208f9ae35885c15a7cf7bdbe161f312"}
Jul 28 17:53:10.798085 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:10.797915    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb" event={"ID":"b33f6041-ae8b-41ef-b608-356bfdbb1e0c","Type":"ContainerStarted","Data":"69a0d60b94df39809221fd1d261b13953428524136a4b0cfd09e4bf5f2ffd635"}
Jul 28 17:53:14.813408 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:14.813374    2581 generic.go:356] "Generic (PLEG): container finished" podID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" containerID="240f5b645ff0e974fdefdb35892ceac43208f9ae35885c15a7cf7bdbe161f312" exitCode=0
Jul 28 17:53:14.813829 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:14.813450    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb" event={"ID":"b33f6041-ae8b-41ef-b608-356bfdbb1e0c","Type":"ContainerDied","Data":"240f5b645ff0e974fdefdb35892ceac43208f9ae35885c15a7cf7bdbe161f312"}
Jul 28 17:53:14.876893 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:14.876877    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:53:15.818759 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:15.818724    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb" event={"ID":"b33f6041-ae8b-41ef-b608-356bfdbb1e0c","Type":"ContainerStarted","Data":"3d789d973dd5a103c4c526e836cdf49fc158f0fa8ecb2505e406bdbb7ea48a06"}
Jul 28 17:53:17.828492 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:17.828400    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb" event={"ID":"b33f6041-ae8b-41ef-b608-356bfdbb1e0c","Type":"ContainerStarted","Data":"4fd80e16cc22c55190b17ff2b77569d0120a565594e368e5255e823df4d7b90c"}
Jul 28 17:53:17.828871 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:17.828498    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb" event={"ID":"b33f6041-ae8b-41ef-b608-356bfdbb1e0c","Type":"ContainerStarted","Data":"d8077558fc7fa5fd5331db4ef7918eb8108b66fe57a6dfb7c0b27cdcc7a22cca"}
Jul 28 17:53:17.828871 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:17.828529    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"
Jul 28 17:53:17.828871 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:17.828563    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"
Jul 28 17:53:17.865400 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:17.865345    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb" podStartSLOduration=2.66574227 podStartE2EDuration="8.865329464s" totalImagesPullingTime="2.607724807s" totalInitContainerRuntime="3.591862387s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:53:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:53:17.864901842 +0000 UTC m=+2646.139518286" watchObservedRunningTime="2026-07-28 17:53:17.865329464 +0000 UTC m=+2646.139945896"
Jul 28 17:53:18.831882 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:18.831853    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"
Jul 28 17:53:24.840181 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:24.840146    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"
Jul 28 17:53:44.840960 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:44.840902    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb" podUID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.40:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 17:53:54.841969 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:53:54.841922    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"
Jul 28 17:54:12.338740 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:12.338668    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 17:54:12.341459 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:12.341439    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 17:54:24.842914 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:24.842886    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"
Jul 28 17:54:29.474474 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:29.474443    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"]
Jul 28 17:54:29.474887 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:29.474740    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb" podUID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" containerName="kserve-container" containerID="cri-o://3d789d973dd5a103c4c526e836cdf49fc158f0fa8ecb2505e406bdbb7ea48a06" gracePeriod=30
Jul 28 17:54:29.474887 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:29.474796    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb" podUID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" containerName="kube-rbac-proxy" containerID="cri-o://4fd80e16cc22c55190b17ff2b77569d0120a565594e368e5255e823df4d7b90c" gracePeriod=30
Jul 28 17:54:29.474887 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:29.474791    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb" podUID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" containerName="kserve-agent" containerID="cri-o://d8077558fc7fa5fd5331db4ef7918eb8108b66fe57a6dfb7c0b27cdcc7a22cca" gracePeriod=30
Jul 28 17:54:29.493869 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:29.493845    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:54:29.835630 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:29.835537    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb" podUID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 28 17:54:30.087792 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:30.087712    2581 generic.go:356] "Generic (PLEG): container finished" podID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" containerID="4fd80e16cc22c55190b17ff2b77569d0120a565594e368e5255e823df4d7b90c" exitCode=2
Jul 28 17:54:30.087792 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:30.087778    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb" event={"ID":"b33f6041-ae8b-41ef-b608-356bfdbb1e0c","Type":"ContainerDied","Data":"4fd80e16cc22c55190b17ff2b77569d0120a565594e368e5255e823df4d7b90c"}
Jul 28 17:54:32.097220 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:32.097189    2581 generic.go:356] "Generic (PLEG): container finished" podID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" containerID="3d789d973dd5a103c4c526e836cdf49fc158f0fa8ecb2505e406bdbb7ea48a06" exitCode=0
Jul 28 17:54:32.097592 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:32.097242    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb" event={"ID":"b33f6041-ae8b-41ef-b608-356bfdbb1e0c","Type":"ContainerDied","Data":"3d789d973dd5a103c4c526e836cdf49fc158f0fa8ecb2505e406bdbb7ea48a06"}
Jul 28 17:54:34.604077 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:34.604037    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:54:34.733270 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:34.733236    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:54:34.835730 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:34.835691    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb" podUID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 28 17:54:34.841075 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:34.841048    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb" podUID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.40:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 17:54:39.835276 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:39.835222    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb" podUID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 28 17:54:39.835761 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:39.835349    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"
Jul 28 17:54:44.836155 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:44.836109    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb" podUID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 28 17:54:44.841572 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:44.841544    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb" podUID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.40:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 17:54:49.836190 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:49.836141    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb" podUID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 28 17:54:54.835603 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:54.835563    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb" podUID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 28 17:54:54.840910 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:54.840880    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb" podUID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.40:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 17:54:54.841034 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:54.841021    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"
Jul 28 17:54:59.641683 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:59.641658    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"
Jul 28 17:54:59.670572 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:59.670546    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-kube-rbac-proxy-sar-config\") pod \"b33f6041-ae8b-41ef-b608-356bfdbb1e0c\" (UID: \"b33f6041-ae8b-41ef-b608-356bfdbb1e0c\") "
Jul 28 17:54:59.670675 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:59.670580    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-proxy-tls\") pod \"b33f6041-ae8b-41ef-b608-356bfdbb1e0c\" (UID: \"b33f6041-ae8b-41ef-b608-356bfdbb1e0c\") "
Jul 28 17:54:59.670675 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:59.670639    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-kserve-provision-location\") pod \"b33f6041-ae8b-41ef-b608-356bfdbb1e0c\" (UID: \"b33f6041-ae8b-41ef-b608-356bfdbb1e0c\") "
Jul 28 17:54:59.670675 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:59.670668    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-cabundle-cert\") pod \"b33f6041-ae8b-41ef-b608-356bfdbb1e0c\" (UID: \"b33f6041-ae8b-41ef-b608-356bfdbb1e0c\") "
Jul 28 17:54:59.670834 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:59.670706    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kw5hd\" (UniqueName: \"kubernetes.io/projected/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-kube-api-access-kw5hd\") pod \"b33f6041-ae8b-41ef-b608-356bfdbb1e0c\" (UID: \"b33f6041-ae8b-41ef-b608-356bfdbb1e0c\") "
Jul 28 17:54:59.671020 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:59.670995    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b33f6041-ae8b-41ef-b608-356bfdbb1e0c" (UID: "b33f6041-ae8b-41ef-b608-356bfdbb1e0c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:54:59.671089 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:59.671010    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b33f6041-ae8b-41ef-b608-356bfdbb1e0c" (UID: "b33f6041-ae8b-41ef-b608-356bfdbb1e0c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:54:59.671127 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:59.671091    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b33f6041-ae8b-41ef-b608-356bfdbb1e0c" (UID: "b33f6041-ae8b-41ef-b608-356bfdbb1e0c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:54:59.672631 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:59.672603    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b33f6041-ae8b-41ef-b608-356bfdbb1e0c" (UID: "b33f6041-ae8b-41ef-b608-356bfdbb1e0c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:54:59.672721 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:59.672698    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-kube-api-access-kw5hd" (OuterVolumeSpecName: "kube-api-access-kw5hd") pod "b33f6041-ae8b-41ef-b608-356bfdbb1e0c" (UID: "b33f6041-ae8b-41ef-b608-356bfdbb1e0c"). InnerVolumeSpecName "kube-api-access-kw5hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:54:59.771431 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:59.771371    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:54:59.771431 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:59.771391    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:54:59.771431 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:59.771403    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:54:59.771431 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:59.771411    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:54:59.771431 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:54:59.771421    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw5hd\" (UniqueName: \"kubernetes.io/projected/b33f6041-ae8b-41ef-b608-356bfdbb1e0c-kube-api-access-kw5hd\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:55:00.191657 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:00.191619    2581 generic.go:356] "Generic (PLEG): container finished" podID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" containerID="d8077558fc7fa5fd5331db4ef7918eb8108b66fe57a6dfb7c0b27cdcc7a22cca" exitCode=137
Jul 28 17:55:00.191869 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:00.191676    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"
Jul 28 17:55:00.191869 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:00.191696    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb" event={"ID":"b33f6041-ae8b-41ef-b608-356bfdbb1e0c","Type":"ContainerDied","Data":"d8077558fc7fa5fd5331db4ef7918eb8108b66fe57a6dfb7c0b27cdcc7a22cca"}
Jul 28 17:55:00.191869 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:00.191778    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb" event={"ID":"b33f6041-ae8b-41ef-b608-356bfdbb1e0c","Type":"ContainerDied","Data":"69a0d60b94df39809221fd1d261b13953428524136a4b0cfd09e4bf5f2ffd635"}
Jul 28 17:55:00.191869 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:00.191821    2581 scope.go:117] "RemoveContainer" containerID="4fd80e16cc22c55190b17ff2b77569d0120a565594e368e5255e823df4d7b90c"
Jul 28 17:55:00.200212 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:00.200104    2581 scope.go:117] "RemoveContainer" containerID="d8077558fc7fa5fd5331db4ef7918eb8108b66fe57a6dfb7c0b27cdcc7a22cca"
Jul 28 17:55:00.207437 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:00.207415    2581 scope.go:117] "RemoveContainer" containerID="3d789d973dd5a103c4c526e836cdf49fc158f0fa8ecb2505e406bdbb7ea48a06"
Jul 28 17:55:00.214062 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:00.214046    2581 scope.go:117] "RemoveContainer" containerID="240f5b645ff0e974fdefdb35892ceac43208f9ae35885c15a7cf7bdbe161f312"
Jul 28 17:55:00.220447 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:00.220431    2581 scope.go:117] "RemoveContainer" containerID="4fd80e16cc22c55190b17ff2b77569d0120a565594e368e5255e823df4d7b90c"
Jul 28 17:55:00.220686 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:55:00.220668    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd80e16cc22c55190b17ff2b77569d0120a565594e368e5255e823df4d7b90c\": container with ID starting with 4fd80e16cc22c55190b17ff2b77569d0120a565594e368e5255e823df4d7b90c not found: ID does not exist" containerID="4fd80e16cc22c55190b17ff2b77569d0120a565594e368e5255e823df4d7b90c"
Jul 28 17:55:00.220745 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:00.220697    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd80e16cc22c55190b17ff2b77569d0120a565594e368e5255e823df4d7b90c"} err="failed to get container status \"4fd80e16cc22c55190b17ff2b77569d0120a565594e368e5255e823df4d7b90c\": rpc error: code = NotFound desc = could not find container \"4fd80e16cc22c55190b17ff2b77569d0120a565594e368e5255e823df4d7b90c\": container with ID starting with 4fd80e16cc22c55190b17ff2b77569d0120a565594e368e5255e823df4d7b90c not found: ID does not exist"
Jul 28 17:55:00.220745 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:00.220713    2581 scope.go:117] "RemoveContainer" containerID="d8077558fc7fa5fd5331db4ef7918eb8108b66fe57a6dfb7c0b27cdcc7a22cca"
Jul 28 17:55:00.220922 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:55:00.220908    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8077558fc7fa5fd5331db4ef7918eb8108b66fe57a6dfb7c0b27cdcc7a22cca\": container with ID starting with d8077558fc7fa5fd5331db4ef7918eb8108b66fe57a6dfb7c0b27cdcc7a22cca not found: ID does not exist" containerID="d8077558fc7fa5fd5331db4ef7918eb8108b66fe57a6dfb7c0b27cdcc7a22cca"
Jul 28 17:55:00.221003 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:00.220926    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8077558fc7fa5fd5331db4ef7918eb8108b66fe57a6dfb7c0b27cdcc7a22cca"} err="failed to get container status \"d8077558fc7fa5fd5331db4ef7918eb8108b66fe57a6dfb7c0b27cdcc7a22cca\": rpc error: code = NotFound desc = could not find container \"d8077558fc7fa5fd5331db4ef7918eb8108b66fe57a6dfb7c0b27cdcc7a22cca\": container with ID starting with d8077558fc7fa5fd5331db4ef7918eb8108b66fe57a6dfb7c0b27cdcc7a22cca not found: ID does not exist"
Jul 28 17:55:00.221003 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:00.220953    2581 scope.go:117] "RemoveContainer" containerID="3d789d973dd5a103c4c526e836cdf49fc158f0fa8ecb2505e406bdbb7ea48a06"
Jul 28 17:55:00.221173 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:55:00.221151    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d789d973dd5a103c4c526e836cdf49fc158f0fa8ecb2505e406bdbb7ea48a06\": container with ID starting with 3d789d973dd5a103c4c526e836cdf49fc158f0fa8ecb2505e406bdbb7ea48a06 not found: ID does not exist" containerID="3d789d973dd5a103c4c526e836cdf49fc158f0fa8ecb2505e406bdbb7ea48a06"
Jul 28 17:55:00.221213 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:00.221180    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d789d973dd5a103c4c526e836cdf49fc158f0fa8ecb2505e406bdbb7ea48a06"} err="failed to get container status \"3d789d973dd5a103c4c526e836cdf49fc158f0fa8ecb2505e406bdbb7ea48a06\": rpc error: code = NotFound desc = could not find container \"3d789d973dd5a103c4c526e836cdf49fc158f0fa8ecb2505e406bdbb7ea48a06\": container with ID starting with 3d789d973dd5a103c4c526e836cdf49fc158f0fa8ecb2505e406bdbb7ea48a06 not found: ID does not exist"
Jul 28 17:55:00.221213 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:00.221197    2581 scope.go:117] "RemoveContainer" containerID="240f5b645ff0e974fdefdb35892ceac43208f9ae35885c15a7cf7bdbe161f312"
Jul 28 17:55:00.221395 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:55:00.221380    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"240f5b645ff0e974fdefdb35892ceac43208f9ae35885c15a7cf7bdbe161f312\": container with ID starting with 240f5b645ff0e974fdefdb35892ceac43208f9ae35885c15a7cf7bdbe161f312 not found: ID does not exist" containerID="240f5b645ff0e974fdefdb35892ceac43208f9ae35885c15a7cf7bdbe161f312"
Jul 28 17:55:00.221477 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:00.221399    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"240f5b645ff0e974fdefdb35892ceac43208f9ae35885c15a7cf7bdbe161f312"} err="failed to get container status \"240f5b645ff0e974fdefdb35892ceac43208f9ae35885c15a7cf7bdbe161f312\": rpc error: code = NotFound desc = could not find container \"240f5b645ff0e974fdefdb35892ceac43208f9ae35885c15a7cf7bdbe161f312\": container with ID starting with 240f5b645ff0e974fdefdb35892ceac43208f9ae35885c15a7cf7bdbe161f312 not found: ID does not exist"
Jul 28 17:55:00.231346 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:00.231325    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"]
Jul 28 17:55:00.236487 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:00.236465    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-64b7f6f8b4-qsmqb"]
Jul 28 17:55:00.289475 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:00.289452    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" path="/var/lib/kubelet/pods/b33f6041-ae8b-41ef-b608-356bfdbb1e0c/volumes"
Jul 28 17:55:23.883473 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:23.883442    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:55:46.387759 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:46.387723    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:55:51.588923 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.588881    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:55:51.672114 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.672081    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"]
Jul 28 17:55:51.672339 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.672327    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" containerName="kserve-container"
Jul 28 17:55:51.672390 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.672340    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" containerName="kserve-container"
Jul 28 17:55:51.672390 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.672349    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" containerName="kserve-agent"
Jul 28 17:55:51.672390 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.672355    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" containerName="kserve-agent"
Jul 28 17:55:51.672390 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.672369    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" containerName="kube-rbac-proxy"
Jul 28 17:55:51.672390 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.672374    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" containerName="kube-rbac-proxy"
Jul 28 17:55:51.672390 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.672390    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" containerName="storage-initializer"
Jul 28 17:55:51.672576 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.672396    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" containerName="storage-initializer"
Jul 28 17:55:51.672576 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.672438    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" containerName="kserve-agent"
Jul 28 17:55:51.672576 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.672444    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" containerName="kube-rbac-proxy"
Jul 28 17:55:51.672576 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.672449    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b33f6041-ae8b-41ef-b608-356bfdbb1e0c" containerName="kserve-container"
Jul 28 17:55:51.675540 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.675516    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"
Jul 28 17:55:51.678656 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.678637    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Jul 28 17:55:51.678768 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.678641    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:55:51.679932 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.679915    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 17:55:51.680168 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.680152    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:55:51.680168 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.680164    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Jul 28 17:55:51.680267 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.680208    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:55:51.680267 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.680232    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:55:51.695744 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.695720    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"]
Jul 28 17:55:51.753140 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.753109    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:55:51.824309 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.824285    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-sh9mf\" (UID: \"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"
Jul 28 17:55:51.824458 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.824324    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-sh9mf\" (UID: \"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"
Jul 28 17:55:51.824458 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.824348    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd7t4\" (UniqueName: \"kubernetes.io/projected/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-kube-api-access-xd7t4\") pod \"isvc-paddle-runtime-predictor-58b99fff76-sh9mf\" (UID: \"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"
Jul 28 17:55:51.824458 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.824365    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-sh9mf\" (UID: \"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"
Jul 28 17:55:51.824458 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.824416    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-sh9mf\" (UID: \"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"
Jul 28 17:55:51.924970 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.924928    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-sh9mf\" (UID: \"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"
Jul 28 17:55:51.925085 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.924987    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-sh9mf\" (UID: \"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"
Jul 28 17:55:51.925085 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.925010    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd7t4\" (UniqueName: \"kubernetes.io/projected/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-kube-api-access-xd7t4\") pod \"isvc-paddle-runtime-predictor-58b99fff76-sh9mf\" (UID: \"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"
Jul 28 17:55:51.925085 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.925028    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-sh9mf\" (UID: \"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"
Jul 28 17:55:51.925085 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.925044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-sh9mf\" (UID: \"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"
Jul 28 17:55:51.925341 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.925320    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-sh9mf\" (UID: \"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"
Jul 28 17:55:51.925693 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.925674    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-sh9mf\" (UID: \"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"
Jul 28 17:55:51.925774 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.925722    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-sh9mf\" (UID: \"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"
Jul 28 17:55:51.927594 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.927570    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-sh9mf\" (UID: \"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"
Jul 28 17:55:51.940090 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.940066    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd7t4\" (UniqueName: \"kubernetes.io/projected/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-kube-api-access-xd7t4\") pod \"isvc-paddle-runtime-predictor-58b99fff76-sh9mf\" (UID: \"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"
Jul 28 17:55:51.985199 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:51.985173    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"
Jul 28 17:55:52.117838 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:52.117808    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"]
Jul 28 17:55:52.123864 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:55:52.123837    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d924bc_c2b6_4f7b_b3da_fbee2fe4a24b.slice/crio-7c54f9bce0652611ec13ffa84b3e92db31d6c8abd7ef84d7b2f1747b625c425d WatchSource:0}: Error finding container 7c54f9bce0652611ec13ffa84b3e92db31d6c8abd7ef84d7b2f1747b625c425d: Status 404 returned error can't find the container with id 7c54f9bce0652611ec13ffa84b3e92db31d6c8abd7ef84d7b2f1747b625c425d
Jul 28 17:55:52.362572 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:52.362477    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf" event={"ID":"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b","Type":"ContainerStarted","Data":"46dc948f4d887c0434f2846d8ec2d3a089b172d2af4de6a00a112bfb7fa93f94"}
Jul 28 17:55:52.362572 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:52.362563    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf" event={"ID":"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b","Type":"ContainerStarted","Data":"7c54f9bce0652611ec13ffa84b3e92db31d6c8abd7ef84d7b2f1747b625c425d"}
Jul 28 17:55:57.380458 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:57.380423    2581 generic.go:356] "Generic (PLEG): container finished" podID="43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b" containerID="46dc948f4d887c0434f2846d8ec2d3a089b172d2af4de6a00a112bfb7fa93f94" exitCode=0
Jul 28 17:55:57.380864 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:55:57.380479    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf" event={"ID":"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b","Type":"ContainerDied","Data":"46dc948f4d887c0434f2846d8ec2d3a089b172d2af4de6a00a112bfb7fa93f94"}
Jul 28 17:56:08.429167 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:56:08.429137    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf" event={"ID":"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b","Type":"ContainerStarted","Data":"ee01894be416c161ed7e0f68c6ec3ec164c75ce862e030925a76b80e562d627e"}
Jul 28 17:56:09.435696 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:56:09.435651    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf" event={"ID":"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b","Type":"ContainerStarted","Data":"7b97e22232738b226911b0ad029dcd78348b07f70d01624b57922990e29fde23"}
Jul 28 17:56:09.436167 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:56:09.435913    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"
Jul 28 17:56:09.458862 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:56:09.458815    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf" podStartSLOduration=2.882650767 podStartE2EDuration="18.458801021s" totalImagesPullingTime="10.953177537s" totalInitContainerRuntime="4.622972717s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:56:09.457990248 +0000 UTC m=+2817.732606680" watchObservedRunningTime="2026-07-28 17:56:09.458801021 +0000 UTC m=+2817.733417455"
Jul 28 17:56:10.439365 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:56:10.439335    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"
Jul 28 17:56:10.440605 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:56:10.440575    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf" podUID="43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 17:56:11.444382 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:56:11.444337    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf" podUID="43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 17:56:16.449095 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:56:16.449059    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"
Jul 28 17:56:16.449635 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:56:16.449604    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf" podUID="43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 17:56:26.450403 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:56:26.450359    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf" podUID="43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 17:56:36.450378 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:56:36.450335    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf" podUID="43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 17:56:46.449649 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:56:46.449609    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf" podUID="43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 17:56:56.450083 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:56:56.450054    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"
Jul 28 17:57:03.239909 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:03.239872    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"]
Jul 28 17:57:03.240825 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:03.240766    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf" podUID="43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b" containerName="kserve-container" containerID="cri-o://ee01894be416c161ed7e0f68c6ec3ec164c75ce862e030925a76b80e562d627e" gracePeriod=30
Jul 28 17:57:03.240977 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:03.240854    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf" podUID="43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b" containerName="kube-rbac-proxy" containerID="cri-o://7b97e22232738b226911b0ad029dcd78348b07f70d01624b57922990e29fde23" gracePeriod=30
Jul 28 17:57:03.498815 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:03.498740    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:57:03.622200 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:03.622167    2581 generic.go:356] "Generic (PLEG): container finished" podID="43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b" containerID="7b97e22232738b226911b0ad029dcd78348b07f70d01624b57922990e29fde23" exitCode=2
Jul 28 17:57:03.622353 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:03.622239    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf" event={"ID":"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b","Type":"ContainerDied","Data":"7b97e22232738b226911b0ad029dcd78348b07f70d01624b57922990e29fde23"}
Jul 28 17:57:05.631865 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:05.631835    2581 generic.go:356] "Generic (PLEG): container finished" podID="43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b" containerID="ee01894be416c161ed7e0f68c6ec3ec164c75ce862e030925a76b80e562d627e" exitCode=0
Jul 28 17:57:05.632302 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:05.631873    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf" event={"ID":"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b","Type":"ContainerDied","Data":"ee01894be416c161ed7e0f68c6ec3ec164c75ce862e030925a76b80e562d627e"}
Jul 28 17:57:05.677970 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:05.677932    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"
Jul 28 17:57:05.752290 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:05.752225    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-kserve-provision-location\") pod \"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b\" (UID: \"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b\") "
Jul 28 17:57:05.752290 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:05.752267    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-proxy-tls\") pod \"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b\" (UID: \"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b\") "
Jul 28 17:57:05.752485 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:05.752302    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xd7t4\" (UniqueName: \"kubernetes.io/projected/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-kube-api-access-xd7t4\") pod \"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b\" (UID: \"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b\") "
Jul 28 17:57:05.752485 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:05.752332    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-cabundle-cert\") pod \"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b\" (UID: \"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b\") "
Jul 28 17:57:05.752485 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:05.752363    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-kube-rbac-proxy-sar-config\") pod \"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b\" (UID: \"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b\") "
Jul 28 17:57:05.752788 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:05.752750    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b" (UID: "43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:57:05.752788 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:05.752765    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b" (UID: "43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:57:05.754353 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:05.754329    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-kube-api-access-xd7t4" (OuterVolumeSpecName: "kube-api-access-xd7t4") pod "43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b" (UID: "43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b"). InnerVolumeSpecName "kube-api-access-xd7t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:57:05.754453 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:05.754329    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b" (UID: "43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:57:05.761450 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:05.761425    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b" (UID: "43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:57:05.853352 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:05.853331    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd7t4\" (UniqueName: \"kubernetes.io/projected/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-kube-api-access-xd7t4\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:57:05.853352 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:05.853350    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:57:05.853474 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:05.853360    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:57:05.853474 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:05.853368    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:57:05.853474 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:05.853377    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:57:06.636775 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:06.636695    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf" event={"ID":"43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b","Type":"ContainerDied","Data":"7c54f9bce0652611ec13ffa84b3e92db31d6c8abd7ef84d7b2f1747b625c425d"}
Jul 28 17:57:06.636775 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:06.636758    2581 scope.go:117] "RemoveContainer" containerID="7b97e22232738b226911b0ad029dcd78348b07f70d01624b57922990e29fde23"
Jul 28 17:57:06.636775 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:06.636710    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"
Jul 28 17:57:06.644010 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:06.643989    2581 scope.go:117] "RemoveContainer" containerID="ee01894be416c161ed7e0f68c6ec3ec164c75ce862e030925a76b80e562d627e"
Jul 28 17:57:06.650849 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:06.650830    2581 scope.go:117] "RemoveContainer" containerID="46dc948f4d887c0434f2846d8ec2d3a089b172d2af4de6a00a112bfb7fa93f94"
Jul 28 17:57:06.665589 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:06.665566    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"]
Jul 28 17:57:06.678296 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:06.678276    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-sh9mf"]
Jul 28 17:57:08.292728 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.292695    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b" path="/var/lib/kubelet/pods/43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b/volumes"
Jul 28 17:57:08.402502 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.402464    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:57:08.458127 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.458098    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"]
Jul 28 17:57:08.458393 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.458380    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b" containerName="kube-rbac-proxy"
Jul 28 17:57:08.458454 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.458395    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b" containerName="kube-rbac-proxy"
Jul 28 17:57:08.458454 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.458404    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b" containerName="storage-initializer"
Jul 28 17:57:08.458454 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.458410    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b" containerName="storage-initializer"
Jul 28 17:57:08.458454 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.458422    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b" containerName="kserve-container"
Jul 28 17:57:08.458454 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.458427    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b" containerName="kserve-container"
Jul 28 17:57:08.458719 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.458468    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b" containerName="kube-rbac-proxy"
Jul 28 17:57:08.458719 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.458475    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="43d924bc-c2b6-4f7b-b3da-fbee2fe4a24b" containerName="kserve-container"
Jul 28 17:57:08.462670 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.462653    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"
Jul 28 17:57:08.465289 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.465267    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:57:08.465464 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.465437    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 17:57:08.465586 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.465489    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Jul 28 17:57:08.465700 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.465688    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:57:08.466563 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.466544    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:57:08.466664 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.466554    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 28 17:57:08.466664 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.466593    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:57:08.472458 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.472436    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"]
Jul 28 17:57:08.573246 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.573167    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g\" (UID: \"a165a6e4-7e10-4fd1-9ac4-7617c292a84b\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"
Jul 28 17:57:08.573246 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.573210    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g\" (UID: \"a165a6e4-7e10-4fd1-9ac4-7617c292a84b\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"
Jul 28 17:57:08.573445 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.573280    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkl8z\" (UniqueName: \"kubernetes.io/projected/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-kube-api-access-vkl8z\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g\" (UID: \"a165a6e4-7e10-4fd1-9ac4-7617c292a84b\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"
Jul 28 17:57:08.573445 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.573320    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g\" (UID: \"a165a6e4-7e10-4fd1-9ac4-7617c292a84b\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"
Jul 28 17:57:08.573445 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.573343    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g\" (UID: \"a165a6e4-7e10-4fd1-9ac4-7617c292a84b\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"
Jul 28 17:57:08.674004 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.673968    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkl8z\" (UniqueName: \"kubernetes.io/projected/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-kube-api-access-vkl8z\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g\" (UID: \"a165a6e4-7e10-4fd1-9ac4-7617c292a84b\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"
Jul 28 17:57:08.674165 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.674011    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g\" (UID: \"a165a6e4-7e10-4fd1-9ac4-7617c292a84b\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"
Jul 28 17:57:08.674165 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.674043    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g\" (UID: \"a165a6e4-7e10-4fd1-9ac4-7617c292a84b\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"
Jul 28 17:57:08.674165 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.674077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g\" (UID: \"a165a6e4-7e10-4fd1-9ac4-7617c292a84b\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"
Jul 28 17:57:08.674165 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.674126    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g\" (UID: \"a165a6e4-7e10-4fd1-9ac4-7617c292a84b\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"
Jul 28 17:57:08.674368 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:57:08.674204    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Jul 28 17:57:08.674368 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:57:08.674268    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-proxy-tls podName:a165a6e4-7e10-4fd1-9ac4-7617c292a84b nodeName:}" failed. No retries permitted until 2026-07-28 17:57:09.174249623 +0000 UTC m=+2877.448866037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g" (UID: "a165a6e4-7e10-4fd1-9ac4-7617c292a84b") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Jul 28 17:57:08.674537 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.674517    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g\" (UID: \"a165a6e4-7e10-4fd1-9ac4-7617c292a84b\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"
Jul 28 17:57:08.674746 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.674729    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g\" (UID: \"a165a6e4-7e10-4fd1-9ac4-7617c292a84b\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"
Jul 28 17:57:08.674857 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.674837    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g\" (UID: \"a165a6e4-7e10-4fd1-9ac4-7617c292a84b\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"
Jul 28 17:57:08.687876 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.687847    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkl8z\" (UniqueName: \"kubernetes.io/projected/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-kube-api-access-vkl8z\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g\" (UID: \"a165a6e4-7e10-4fd1-9ac4-7617c292a84b\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"
Jul 28 17:57:08.909338 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:08.909313    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:57:09.178746 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:09.178670    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g\" (UID: \"a165a6e4-7e10-4fd1-9ac4-7617c292a84b\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"
Jul 28 17:57:09.180920 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:09.180893    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g\" (UID: \"a165a6e4-7e10-4fd1-9ac4-7617c292a84b\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"
Jul 28 17:57:09.373992 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:09.373962    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"
Jul 28 17:57:09.505083 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:09.505055    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"]
Jul 28 17:57:09.508622 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:57:09.508593    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda165a6e4_7e10_4fd1_9ac4_7617c292a84b.slice/crio-8cde97c1cf23d2adffaca1ad00c53418d453c4969bf937de0f8879f017c32f1c WatchSource:0}: Error finding container 8cde97c1cf23d2adffaca1ad00c53418d453c4969bf937de0f8879f017c32f1c: Status 404 returned error can't find the container with id 8cde97c1cf23d2adffaca1ad00c53418d453c4969bf937de0f8879f017c32f1c
Jul 28 17:57:09.650515 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:09.650483    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g" event={"ID":"a165a6e4-7e10-4fd1-9ac4-7617c292a84b","Type":"ContainerStarted","Data":"fea1b78a9478e12c3e8bc9fe1d1dd56b36ba2cb0320279a6b59ba25a6215f197"}
Jul 28 17:57:09.650710 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:09.650548    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g" event={"ID":"a165a6e4-7e10-4fd1-9ac4-7617c292a84b","Type":"ContainerStarted","Data":"8cde97c1cf23d2adffaca1ad00c53418d453c4969bf937de0f8879f017c32f1c"}
Jul 28 17:57:14.669446 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:14.669412    2581 generic.go:356] "Generic (PLEG): container finished" podID="a165a6e4-7e10-4fd1-9ac4-7617c292a84b" containerID="fea1b78a9478e12c3e8bc9fe1d1dd56b36ba2cb0320279a6b59ba25a6215f197" exitCode=0
Jul 28 17:57:14.669841 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:14.669492    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g" event={"ID":"a165a6e4-7e10-4fd1-9ac4-7617c292a84b","Type":"ContainerDied","Data":"fea1b78a9478e12c3e8bc9fe1d1dd56b36ba2cb0320279a6b59ba25a6215f197"}
Jul 28 17:57:15.675096 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:15.675062    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g" event={"ID":"a165a6e4-7e10-4fd1-9ac4-7617c292a84b","Type":"ContainerStarted","Data":"b4258fc6025f3f011500ed4b77fc7f493e3d37a786857e76d8e0e9b8dea59588"}
Jul 28 17:57:15.675478 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:15.675128    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g" event={"ID":"a165a6e4-7e10-4fd1-9ac4-7617c292a84b","Type":"ContainerStarted","Data":"070802718c6da139ffcc1f78d8dab7712c65396ea60bebed5ebc0dd60a85444d"}
Jul 28 17:57:15.675478 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:15.675334    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"
Jul 28 17:57:15.699048 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:15.698996    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g" podStartSLOduration=3.325020177 podStartE2EDuration="7.698982871s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.373962694s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:15.697976325 +0000 UTC m=+2883.972592757" watchObservedRunningTime="2026-07-28 17:57:15.698982871 +0000 UTC m=+2883.973599306"
Jul 28 17:57:16.678443 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:16.678412    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"
Jul 28 17:57:16.679660 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:16.679636    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g" podUID="a165a6e4-7e10-4fd1-9ac4-7617c292a84b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 17:57:17.682701 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:17.682661    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g" podUID="a165a6e4-7e10-4fd1-9ac4-7617c292a84b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 17:57:22.687240 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:22.687207    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"
Jul 28 17:57:22.687794 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:22.687767    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g" podUID="a165a6e4-7e10-4fd1-9ac4-7617c292a84b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 17:57:32.688441 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:32.688401    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g" podUID="a165a6e4-7e10-4fd1-9ac4-7617c292a84b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 17:57:42.688355 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:42.688307    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g" podUID="a165a6e4-7e10-4fd1-9ac4-7617c292a84b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 17:57:52.688438 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:57:52.688387    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g" podUID="a165a6e4-7e10-4fd1-9ac4-7617c292a84b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 17:58:02.688550 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:02.688520    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"
Jul 28 17:58:10.256523 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:10.256488    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"]
Jul 28 17:58:10.256930 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:10.256722    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g" podUID="a165a6e4-7e10-4fd1-9ac4-7617c292a84b" containerName="kserve-container" containerID="cri-o://070802718c6da139ffcc1f78d8dab7712c65396ea60bebed5ebc0dd60a85444d" gracePeriod=30
Jul 28 17:58:10.256930 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:10.256767    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g" podUID="a165a6e4-7e10-4fd1-9ac4-7617c292a84b" containerName="kube-rbac-proxy" containerID="cri-o://b4258fc6025f3f011500ed4b77fc7f493e3d37a786857e76d8e0e9b8dea59588" gracePeriod=30
Jul 28 17:58:10.268644 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:10.268616    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:58:10.862111 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:10.862079    2581 generic.go:356] "Generic (PLEG): container finished" podID="a165a6e4-7e10-4fd1-9ac4-7617c292a84b" containerID="b4258fc6025f3f011500ed4b77fc7f493e3d37a786857e76d8e0e9b8dea59588" exitCode=2
Jul 28 17:58:10.862286 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:10.862128    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g" event={"ID":"a165a6e4-7e10-4fd1-9ac4-7617c292a84b","Type":"ContainerDied","Data":"b4258fc6025f3f011500ed4b77fc7f493e3d37a786857e76d8e0e9b8dea59588"}
Jul 28 17:58:12.683106 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:12.683053    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g" podUID="a165a6e4-7e10-4fd1-9ac4-7617c292a84b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Jul 28 17:58:12.688478 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:12.688440    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g" podUID="a165a6e4-7e10-4fd1-9ac4-7617c292a84b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 28 17:58:12.802434 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:12.802407    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"
Jul 28 17:58:12.875352 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:12.875315    2581 generic.go:356] "Generic (PLEG): container finished" podID="a165a6e4-7e10-4fd1-9ac4-7617c292a84b" containerID="070802718c6da139ffcc1f78d8dab7712c65396ea60bebed5ebc0dd60a85444d" exitCode=0
Jul 28 17:58:12.875536 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:12.875361    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g" event={"ID":"a165a6e4-7e10-4fd1-9ac4-7617c292a84b","Type":"ContainerDied","Data":"070802718c6da139ffcc1f78d8dab7712c65396ea60bebed5ebc0dd60a85444d"}
Jul 28 17:58:12.875536 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:12.875392    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"
Jul 28 17:58:12.875536 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:12.875477    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g" event={"ID":"a165a6e4-7e10-4fd1-9ac4-7617c292a84b","Type":"ContainerDied","Data":"8cde97c1cf23d2adffaca1ad00c53418d453c4969bf937de0f8879f017c32f1c"}
Jul 28 17:58:12.875536 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:12.875519    2581 scope.go:117] "RemoveContainer" containerID="b4258fc6025f3f011500ed4b77fc7f493e3d37a786857e76d8e0e9b8dea59588"
Jul 28 17:58:12.884188 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:12.884173    2581 scope.go:117] "RemoveContainer" containerID="070802718c6da139ffcc1f78d8dab7712c65396ea60bebed5ebc0dd60a85444d"
Jul 28 17:58:12.890901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:12.890885    2581 scope.go:117] "RemoveContainer" containerID="fea1b78a9478e12c3e8bc9fe1d1dd56b36ba2cb0320279a6b59ba25a6215f197"
Jul 28 17:58:12.897186 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:12.897171    2581 scope.go:117] "RemoveContainer" containerID="b4258fc6025f3f011500ed4b77fc7f493e3d37a786857e76d8e0e9b8dea59588"
Jul 28 17:58:12.897421 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:58:12.897403    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4258fc6025f3f011500ed4b77fc7f493e3d37a786857e76d8e0e9b8dea59588\": container with ID starting with b4258fc6025f3f011500ed4b77fc7f493e3d37a786857e76d8e0e9b8dea59588 not found: ID does not exist" containerID="b4258fc6025f3f011500ed4b77fc7f493e3d37a786857e76d8e0e9b8dea59588"
Jul 28 17:58:12.897474 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:12.897432    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4258fc6025f3f011500ed4b77fc7f493e3d37a786857e76d8e0e9b8dea59588"} err="failed to get container status \"b4258fc6025f3f011500ed4b77fc7f493e3d37a786857e76d8e0e9b8dea59588\": rpc error: code = NotFound desc = could not find container \"b4258fc6025f3f011500ed4b77fc7f493e3d37a786857e76d8e0e9b8dea59588\": container with ID starting with b4258fc6025f3f011500ed4b77fc7f493e3d37a786857e76d8e0e9b8dea59588 not found: ID does not exist"
Jul 28 17:58:12.897474 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:12.897449    2581 scope.go:117] "RemoveContainer" containerID="070802718c6da139ffcc1f78d8dab7712c65396ea60bebed5ebc0dd60a85444d"
Jul 28 17:58:12.897674 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:58:12.897659    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"070802718c6da139ffcc1f78d8dab7712c65396ea60bebed5ebc0dd60a85444d\": container with ID starting with 070802718c6da139ffcc1f78d8dab7712c65396ea60bebed5ebc0dd60a85444d not found: ID does not exist" containerID="070802718c6da139ffcc1f78d8dab7712c65396ea60bebed5ebc0dd60a85444d"
Jul 28 17:58:12.897717 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:12.897685    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"070802718c6da139ffcc1f78d8dab7712c65396ea60bebed5ebc0dd60a85444d"} err="failed to get container status \"070802718c6da139ffcc1f78d8dab7712c65396ea60bebed5ebc0dd60a85444d\": rpc error: code = NotFound desc = could not find container \"070802718c6da139ffcc1f78d8dab7712c65396ea60bebed5ebc0dd60a85444d\": container with ID starting with 070802718c6da139ffcc1f78d8dab7712c65396ea60bebed5ebc0dd60a85444d not found: ID does not exist"
Jul 28 17:58:12.897717 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:12.897698    2581 scope.go:117] "RemoveContainer" containerID="fea1b78a9478e12c3e8bc9fe1d1dd56b36ba2cb0320279a6b59ba25a6215f197"
Jul 28 17:58:12.897889 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:58:12.897874    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fea1b78a9478e12c3e8bc9fe1d1dd56b36ba2cb0320279a6b59ba25a6215f197\": container with ID starting with fea1b78a9478e12c3e8bc9fe1d1dd56b36ba2cb0320279a6b59ba25a6215f197 not found: ID does not exist" containerID="fea1b78a9478e12c3e8bc9fe1d1dd56b36ba2cb0320279a6b59ba25a6215f197"
Jul 28 17:58:12.897929 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:12.897893    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fea1b78a9478e12c3e8bc9fe1d1dd56b36ba2cb0320279a6b59ba25a6215f197"} err="failed to get container status \"fea1b78a9478e12c3e8bc9fe1d1dd56b36ba2cb0320279a6b59ba25a6215f197\": rpc error: code = NotFound desc = could not find container \"fea1b78a9478e12c3e8bc9fe1d1dd56b36ba2cb0320279a6b59ba25a6215f197\": container with ID starting with fea1b78a9478e12c3e8bc9fe1d1dd56b36ba2cb0320279a6b59ba25a6215f197 not found: ID does not exist"
Jul 28 17:58:12.939668 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:12.939616    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-cabundle-cert\") pod \"a165a6e4-7e10-4fd1-9ac4-7617c292a84b\" (UID: \"a165a6e4-7e10-4fd1-9ac4-7617c292a84b\") "
Jul 28 17:58:12.939748 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:12.939670    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-kserve-provision-location\") pod \"a165a6e4-7e10-4fd1-9ac4-7617c292a84b\" (UID: \"a165a6e4-7e10-4fd1-9ac4-7617c292a84b\") "
Jul 28 17:58:12.939748 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:12.939709    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-kube-rbac-proxy-sar-config\") pod \"a165a6e4-7e10-4fd1-9ac4-7617c292a84b\" (UID: \"a165a6e4-7e10-4fd1-9ac4-7617c292a84b\") "
Jul 28 17:58:12.939748 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:12.939743    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-proxy-tls\") pod \"a165a6e4-7e10-4fd1-9ac4-7617c292a84b\" (UID: \"a165a6e4-7e10-4fd1-9ac4-7617c292a84b\") "
Jul 28 17:58:12.939904 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:12.939769    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkl8z\" (UniqueName: \"kubernetes.io/projected/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-kube-api-access-vkl8z\") pod \"a165a6e4-7e10-4fd1-9ac4-7617c292a84b\" (UID: \"a165a6e4-7e10-4fd1-9ac4-7617c292a84b\") "
Jul 28 17:58:12.940078 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:12.940058    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a165a6e4-7e10-4fd1-9ac4-7617c292a84b" (UID: "a165a6e4-7e10-4fd1-9ac4-7617c292a84b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:58:12.940151 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:12.940133    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a165a6e4-7e10-4fd1-9ac4-7617c292a84b" (UID: "a165a6e4-7e10-4fd1-9ac4-7617c292a84b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:58:12.941666 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:12.941639    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a165a6e4-7e10-4fd1-9ac4-7617c292a84b" (UID: "a165a6e4-7e10-4fd1-9ac4-7617c292a84b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:58:12.941787 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:12.941767    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-kube-api-access-vkl8z" (OuterVolumeSpecName: "kube-api-access-vkl8z") pod "a165a6e4-7e10-4fd1-9ac4-7617c292a84b" (UID: "a165a6e4-7e10-4fd1-9ac4-7617c292a84b"). InnerVolumeSpecName "kube-api-access-vkl8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:58:12.949557 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:12.949534    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a165a6e4-7e10-4fd1-9ac4-7617c292a84b" (UID: "a165a6e4-7e10-4fd1-9ac4-7617c292a84b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:58:13.040402 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:13.040372    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:58:13.040402 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:13.040400    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:58:13.040539 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:13.040414    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:58:13.040539 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:13.040427    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkl8z\" (UniqueName: \"kubernetes.io/projected/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-kube-api-access-vkl8z\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:58:13.040539 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:13.040439    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a165a6e4-7e10-4fd1-9ac4-7617c292a84b-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:58:13.202823 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:13.202749    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"]
Jul 28 17:58:13.207733 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:13.207709    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-zds6g"]
Jul 28 17:58:14.289675 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:14.289640    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a165a6e4-7e10-4fd1-9ac4-7617c292a84b" path="/var/lib/kubelet/pods/a165a6e4-7e10-4fd1-9ac4-7617c292a84b/volumes"
Jul 28 17:58:15.375847 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.375817    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:58:15.447308 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.447282    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"]
Jul 28 17:58:15.447535 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.447523    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a165a6e4-7e10-4fd1-9ac4-7617c292a84b" containerName="kserve-container"
Jul 28 17:58:15.447582 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.447537    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a165a6e4-7e10-4fd1-9ac4-7617c292a84b" containerName="kserve-container"
Jul 28 17:58:15.447582 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.447546    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a165a6e4-7e10-4fd1-9ac4-7617c292a84b" containerName="storage-initializer"
Jul 28 17:58:15.447582 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.447552    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a165a6e4-7e10-4fd1-9ac4-7617c292a84b" containerName="storage-initializer"
Jul 28 17:58:15.447582 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.447559    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a165a6e4-7e10-4fd1-9ac4-7617c292a84b" containerName="kube-rbac-proxy"
Jul 28 17:58:15.447582 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.447564    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a165a6e4-7e10-4fd1-9ac4-7617c292a84b" containerName="kube-rbac-proxy"
Jul 28 17:58:15.447741 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.447620    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a165a6e4-7e10-4fd1-9ac4-7617c292a84b" containerName="kserve-container"
Jul 28 17:58:15.447741 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.447628    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a165a6e4-7e10-4fd1-9ac4-7617c292a84b" containerName="kube-rbac-proxy"
Jul 28 17:58:15.452056 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.452036    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"
Jul 28 17:58:15.454963 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.454928    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:58:15.455130 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.455111    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 17:58:15.455199 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.455185    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:58:15.456317 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.456300    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:58:15.456424 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.456302    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Jul 28 17:58:15.456424 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.456303    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Jul 28 17:58:15.456424 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.456350    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:58:15.462516 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.462494    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"]
Jul 28 17:58:15.487104 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.487084    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:58:15.556720 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.556700    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/660e7892-5897-409d-8080-2fb9977fdf67-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-qjbkj\" (UID: \"660e7892-5897-409d-8080-2fb9977fdf67\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"
Jul 28 17:58:15.556829 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.556731    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/660e7892-5897-409d-8080-2fb9977fdf67-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-qjbkj\" (UID: \"660e7892-5897-409d-8080-2fb9977fdf67\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"
Jul 28 17:58:15.556829 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.556755    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gqrt\" (UniqueName: \"kubernetes.io/projected/660e7892-5897-409d-8080-2fb9977fdf67-kube-api-access-7gqrt\") pod \"isvc-pmml-predictor-8674795f59-qjbkj\" (UID: \"660e7892-5897-409d-8080-2fb9977fdf67\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"
Jul 28 17:58:15.556829 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.556805    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/660e7892-5897-409d-8080-2fb9977fdf67-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-qjbkj\" (UID: \"660e7892-5897-409d-8080-2fb9977fdf67\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"
Jul 28 17:58:15.557006 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.556845    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/660e7892-5897-409d-8080-2fb9977fdf67-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-qjbkj\" (UID: \"660e7892-5897-409d-8080-2fb9977fdf67\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"
Jul 28 17:58:15.658177 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.658151    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/660e7892-5897-409d-8080-2fb9977fdf67-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-qjbkj\" (UID: \"660e7892-5897-409d-8080-2fb9977fdf67\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"
Jul 28 17:58:15.658304 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.658190    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/660e7892-5897-409d-8080-2fb9977fdf67-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-qjbkj\" (UID: \"660e7892-5897-409d-8080-2fb9977fdf67\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"
Jul 28 17:58:15.658304 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.658216    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gqrt\" (UniqueName: \"kubernetes.io/projected/660e7892-5897-409d-8080-2fb9977fdf67-kube-api-access-7gqrt\") pod \"isvc-pmml-predictor-8674795f59-qjbkj\" (UID: \"660e7892-5897-409d-8080-2fb9977fdf67\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"
Jul 28 17:58:15.658304 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.658247    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/660e7892-5897-409d-8080-2fb9977fdf67-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-qjbkj\" (UID: \"660e7892-5897-409d-8080-2fb9977fdf67\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"
Jul 28 17:58:15.658304 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.658274    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/660e7892-5897-409d-8080-2fb9977fdf67-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-qjbkj\" (UID: \"660e7892-5897-409d-8080-2fb9977fdf67\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"
Jul 28 17:58:15.658724 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.658704    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/660e7892-5897-409d-8080-2fb9977fdf67-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-qjbkj\" (UID: \"660e7892-5897-409d-8080-2fb9977fdf67\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"
Jul 28 17:58:15.659047 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.659023    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/660e7892-5897-409d-8080-2fb9977fdf67-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-qjbkj\" (UID: \"660e7892-5897-409d-8080-2fb9977fdf67\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"
Jul 28 17:58:15.659133 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.659023    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/660e7892-5897-409d-8080-2fb9977fdf67-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-qjbkj\" (UID: \"660e7892-5897-409d-8080-2fb9977fdf67\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"
Jul 28 17:58:15.660735 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.660710    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/660e7892-5897-409d-8080-2fb9977fdf67-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-qjbkj\" (UID: \"660e7892-5897-409d-8080-2fb9977fdf67\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"
Jul 28 17:58:15.667389 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.667361    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gqrt\" (UniqueName: \"kubernetes.io/projected/660e7892-5897-409d-8080-2fb9977fdf67-kube-api-access-7gqrt\") pod \"isvc-pmml-predictor-8674795f59-qjbkj\" (UID: \"660e7892-5897-409d-8080-2fb9977fdf67\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"
Jul 28 17:58:15.762234 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.762209    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"
Jul 28 17:58:15.888742 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:15.888625    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"]
Jul 28 17:58:15.891201 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:58:15.891176    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660e7892_5897_409d_8080_2fb9977fdf67.slice/crio-6fe360400be545f236874682b27960e6849384652d76fe43d70f51823d5c3125 WatchSource:0}: Error finding container 6fe360400be545f236874682b27960e6849384652d76fe43d70f51823d5c3125: Status 404 returned error can't find the container with id 6fe360400be545f236874682b27960e6849384652d76fe43d70f51823d5c3125
Jul 28 17:58:16.890916 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:16.890875    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj" event={"ID":"660e7892-5897-409d-8080-2fb9977fdf67","Type":"ContainerStarted","Data":"8e323fb0a0bb7e773fe3080765fe0e3cfb9f0c07558e14c1de9cd8ae75bbd45d"}
Jul 28 17:58:16.891321 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:16.890961    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj" event={"ID":"660e7892-5897-409d-8080-2fb9977fdf67","Type":"ContainerStarted","Data":"6fe360400be545f236874682b27960e6849384652d76fe43d70f51823d5c3125"}
Jul 28 17:58:19.902330 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:19.902290    2581 generic.go:356] "Generic (PLEG): container finished" podID="660e7892-5897-409d-8080-2fb9977fdf67" containerID="8e323fb0a0bb7e773fe3080765fe0e3cfb9f0c07558e14c1de9cd8ae75bbd45d" exitCode=0
Jul 28 17:58:19.902672 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:19.902336    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj" event={"ID":"660e7892-5897-409d-8080-2fb9977fdf67","Type":"ContainerDied","Data":"8e323fb0a0bb7e773fe3080765fe0e3cfb9f0c07558e14c1de9cd8ae75bbd45d"}
Jul 28 17:58:19.903604 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:19.903586    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:58:23.471741 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:23.471662    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:58:26.936240 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:26.936205    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj" event={"ID":"660e7892-5897-409d-8080-2fb9977fdf67","Type":"ContainerStarted","Data":"55353f99d8087719aeb1c0fe2b0cea9e9ac11ef57454dd28af8452779946ca5e"}
Jul 28 17:58:26.936635 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:26.936269    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj" event={"ID":"660e7892-5897-409d-8080-2fb9977fdf67","Type":"ContainerStarted","Data":"0b0940fdeabebb4ff7773464c5e915576b15e65a0ab9e509917ee85c190ec5d7"}
Jul 28 17:58:26.936635 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:26.936463    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"
Jul 28 17:58:26.960114 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:26.960063    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj" podStartSLOduration=1.422716519 podStartE2EDuration="11.96005097s" totalImagesPullingTime="6.71271225s" totalInitContainerRuntime="3.824622201s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:58:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:58:26.958736724 +0000 UTC m=+2955.233353158" watchObservedRunningTime="2026-07-28 17:58:26.96005097 +0000 UTC m=+2955.234667402"
Jul 28 17:58:27.939690 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:27.939652    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"
Jul 28 17:58:27.940873 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:27.940844    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj" podUID="660e7892-5897-409d-8080-2fb9977fdf67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 17:58:28.943061 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:28.943024    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj" podUID="660e7892-5897-409d-8080-2fb9977fdf67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 17:58:33.947765 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:33.947731    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"
Jul 28 17:58:33.948299 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:33.948272    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj" podUID="660e7892-5897-409d-8080-2fb9977fdf67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 17:58:43.948558 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:43.948516    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj" podUID="660e7892-5897-409d-8080-2fb9977fdf67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 17:58:53.948554 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:58:53.948515    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj" podUID="660e7892-5897-409d-8080-2fb9977fdf67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 17:59:03.948668 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:03.948628    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj" podUID="660e7892-5897-409d-8080-2fb9977fdf67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 17:59:12.358961 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:12.358905    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 17:59:12.361706 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:12.361687    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 17:59:13.948729 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:13.948693    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj" podUID="660e7892-5897-409d-8080-2fb9977fdf67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 17:59:23.948819 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:23.948782    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj" podUID="660e7892-5897-409d-8080-2fb9977fdf67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 17:59:33.948446 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:33.948403    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj" podUID="660e7892-5897-409d-8080-2fb9977fdf67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 17:59:43.948989 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:43.948931    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"
Jul 28 17:59:46.430155 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:46.430126    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"]
Jul 28 17:59:46.430544 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:46.430332    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj" podUID="660e7892-5897-409d-8080-2fb9977fdf67" containerName="kserve-container" containerID="cri-o://0b0940fdeabebb4ff7773464c5e915576b15e65a0ab9e509917ee85c190ec5d7" gracePeriod=30
Jul 28 17:59:46.430544 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:46.430375    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj" podUID="660e7892-5897-409d-8080-2fb9977fdf67" containerName="kube-rbac-proxy" containerID="cri-o://55353f99d8087719aeb1c0fe2b0cea9e9ac11ef57454dd28af8452779946ca5e" gracePeriod=30
Jul 28 17:59:46.442637 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:46.442612    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:59:47.209352 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:47.209320    2581 generic.go:356] "Generic (PLEG): container finished" podID="660e7892-5897-409d-8080-2fb9977fdf67" containerID="55353f99d8087719aeb1c0fe2b0cea9e9ac11ef57454dd28af8452779946ca5e" exitCode=2
Jul 28 17:59:47.209529 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:47.209401    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj" event={"ID":"660e7892-5897-409d-8080-2fb9977fdf67","Type":"ContainerDied","Data":"55353f99d8087719aeb1c0fe2b0cea9e9ac11ef57454dd28af8452779946ca5e"}
Jul 28 17:59:48.943929 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:48.943881    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj" podUID="660e7892-5897-409d-8080-2fb9977fdf67" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Jul 28 17:59:49.970713 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:49.970689    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"
Jul 28 17:59:50.146124 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.146026    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gqrt\" (UniqueName: \"kubernetes.io/projected/660e7892-5897-409d-8080-2fb9977fdf67-kube-api-access-7gqrt\") pod \"660e7892-5897-409d-8080-2fb9977fdf67\" (UID: \"660e7892-5897-409d-8080-2fb9977fdf67\") "
Jul 28 17:59:50.146124 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.146119    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/660e7892-5897-409d-8080-2fb9977fdf67-cabundle-cert\") pod \"660e7892-5897-409d-8080-2fb9977fdf67\" (UID: \"660e7892-5897-409d-8080-2fb9977fdf67\") "
Jul 28 17:59:50.146355 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.146153    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/660e7892-5897-409d-8080-2fb9977fdf67-proxy-tls\") pod \"660e7892-5897-409d-8080-2fb9977fdf67\" (UID: \"660e7892-5897-409d-8080-2fb9977fdf67\") "
Jul 28 17:59:50.146355 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.146195    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/660e7892-5897-409d-8080-2fb9977fdf67-kube-rbac-proxy-sar-config\") pod \"660e7892-5897-409d-8080-2fb9977fdf67\" (UID: \"660e7892-5897-409d-8080-2fb9977fdf67\") "
Jul 28 17:59:50.146355 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.146217    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/660e7892-5897-409d-8080-2fb9977fdf67-kserve-provision-location\") pod \"660e7892-5897-409d-8080-2fb9977fdf67\" (UID: \"660e7892-5897-409d-8080-2fb9977fdf67\") "
Jul 28 17:59:50.146539 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.146514    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660e7892-5897-409d-8080-2fb9977fdf67-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "660e7892-5897-409d-8080-2fb9977fdf67" (UID: "660e7892-5897-409d-8080-2fb9977fdf67"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:59:50.146598 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.146566    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660e7892-5897-409d-8080-2fb9977fdf67-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "660e7892-5897-409d-8080-2fb9977fdf67" (UID: "660e7892-5897-409d-8080-2fb9977fdf67"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:59:50.146651 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.146573    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660e7892-5897-409d-8080-2fb9977fdf67-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "660e7892-5897-409d-8080-2fb9977fdf67" (UID: "660e7892-5897-409d-8080-2fb9977fdf67"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:59:50.148153 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.148130    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660e7892-5897-409d-8080-2fb9977fdf67-kube-api-access-7gqrt" (OuterVolumeSpecName: "kube-api-access-7gqrt") pod "660e7892-5897-409d-8080-2fb9977fdf67" (UID: "660e7892-5897-409d-8080-2fb9977fdf67"). InnerVolumeSpecName "kube-api-access-7gqrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:59:50.148267 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.148231    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/660e7892-5897-409d-8080-2fb9977fdf67-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "660e7892-5897-409d-8080-2fb9977fdf67" (UID: "660e7892-5897-409d-8080-2fb9977fdf67"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:59:50.220546 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.220517    2581 generic.go:356] "Generic (PLEG): container finished" podID="660e7892-5897-409d-8080-2fb9977fdf67" containerID="0b0940fdeabebb4ff7773464c5e915576b15e65a0ab9e509917ee85c190ec5d7" exitCode=0
Jul 28 17:59:50.220687 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.220548    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj" event={"ID":"660e7892-5897-409d-8080-2fb9977fdf67","Type":"ContainerDied","Data":"0b0940fdeabebb4ff7773464c5e915576b15e65a0ab9e509917ee85c190ec5d7"}
Jul 28 17:59:50.220687 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.220587    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"
Jul 28 17:59:50.220687 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.220596    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj" event={"ID":"660e7892-5897-409d-8080-2fb9977fdf67","Type":"ContainerDied","Data":"6fe360400be545f236874682b27960e6849384652d76fe43d70f51823d5c3125"}
Jul 28 17:59:50.220687 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.220622    2581 scope.go:117] "RemoveContainer" containerID="55353f99d8087719aeb1c0fe2b0cea9e9ac11ef57454dd28af8452779946ca5e"
Jul 28 17:59:50.230334 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.230293    2581 scope.go:117] "RemoveContainer" containerID="0b0940fdeabebb4ff7773464c5e915576b15e65a0ab9e509917ee85c190ec5d7"
Jul 28 17:59:50.237434 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.237419    2581 scope.go:117] "RemoveContainer" containerID="8e323fb0a0bb7e773fe3080765fe0e3cfb9f0c07558e14c1de9cd8ae75bbd45d"
Jul 28 17:59:50.244119 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.244102    2581 scope.go:117] "RemoveContainer" containerID="55353f99d8087719aeb1c0fe2b0cea9e9ac11ef57454dd28af8452779946ca5e"
Jul 28 17:59:50.244376 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:59:50.244350    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55353f99d8087719aeb1c0fe2b0cea9e9ac11ef57454dd28af8452779946ca5e\": container with ID starting with 55353f99d8087719aeb1c0fe2b0cea9e9ac11ef57454dd28af8452779946ca5e not found: ID does not exist" containerID="55353f99d8087719aeb1c0fe2b0cea9e9ac11ef57454dd28af8452779946ca5e"
Jul 28 17:59:50.244452 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.244388    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55353f99d8087719aeb1c0fe2b0cea9e9ac11ef57454dd28af8452779946ca5e"} err="failed to get container status \"55353f99d8087719aeb1c0fe2b0cea9e9ac11ef57454dd28af8452779946ca5e\": rpc error: code = NotFound desc = could not find container \"55353f99d8087719aeb1c0fe2b0cea9e9ac11ef57454dd28af8452779946ca5e\": container with ID starting with 55353f99d8087719aeb1c0fe2b0cea9e9ac11ef57454dd28af8452779946ca5e not found: ID does not exist"
Jul 28 17:59:50.244452 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.244411    2581 scope.go:117] "RemoveContainer" containerID="0b0940fdeabebb4ff7773464c5e915576b15e65a0ab9e509917ee85c190ec5d7"
Jul 28 17:59:50.244644 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:59:50.244626    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b0940fdeabebb4ff7773464c5e915576b15e65a0ab9e509917ee85c190ec5d7\": container with ID starting with 0b0940fdeabebb4ff7773464c5e915576b15e65a0ab9e509917ee85c190ec5d7 not found: ID does not exist" containerID="0b0940fdeabebb4ff7773464c5e915576b15e65a0ab9e509917ee85c190ec5d7"
Jul 28 17:59:50.244709 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.244653    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b0940fdeabebb4ff7773464c5e915576b15e65a0ab9e509917ee85c190ec5d7"} err="failed to get container status \"0b0940fdeabebb4ff7773464c5e915576b15e65a0ab9e509917ee85c190ec5d7\": rpc error: code = NotFound desc = could not find container \"0b0940fdeabebb4ff7773464c5e915576b15e65a0ab9e509917ee85c190ec5d7\": container with ID starting with 0b0940fdeabebb4ff7773464c5e915576b15e65a0ab9e509917ee85c190ec5d7 not found: ID does not exist"
Jul 28 17:59:50.244709 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.244675    2581 scope.go:117] "RemoveContainer" containerID="8e323fb0a0bb7e773fe3080765fe0e3cfb9f0c07558e14c1de9cd8ae75bbd45d"
Jul 28 17:59:50.244908 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:59:50.244889    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e323fb0a0bb7e773fe3080765fe0e3cfb9f0c07558e14c1de9cd8ae75bbd45d\": container with ID starting with 8e323fb0a0bb7e773fe3080765fe0e3cfb9f0c07558e14c1de9cd8ae75bbd45d not found: ID does not exist" containerID="8e323fb0a0bb7e773fe3080765fe0e3cfb9f0c07558e14c1de9cd8ae75bbd45d"
Jul 28 17:59:50.244963 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.244914    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e323fb0a0bb7e773fe3080765fe0e3cfb9f0c07558e14c1de9cd8ae75bbd45d"} err="failed to get container status \"8e323fb0a0bb7e773fe3080765fe0e3cfb9f0c07558e14c1de9cd8ae75bbd45d\": rpc error: code = NotFound desc = could not find container \"8e323fb0a0bb7e773fe3080765fe0e3cfb9f0c07558e14c1de9cd8ae75bbd45d\": container with ID starting with 8e323fb0a0bb7e773fe3080765fe0e3cfb9f0c07558e14c1de9cd8ae75bbd45d not found: ID does not exist"
Jul 28 17:59:50.247217 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.247199    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/660e7892-5897-409d-8080-2fb9977fdf67-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:59:50.247273 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.247219    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/660e7892-5897-409d-8080-2fb9977fdf67-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:59:50.247273 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.247229    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gqrt\" (UniqueName: \"kubernetes.io/projected/660e7892-5897-409d-8080-2fb9977fdf67-kube-api-access-7gqrt\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:59:50.247273 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.247240    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/660e7892-5897-409d-8080-2fb9977fdf67-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:59:50.247273 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.247249    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/660e7892-5897-409d-8080-2fb9977fdf67-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 17:59:50.251433 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.251412    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"]
Jul 28 17:59:50.254713 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.254693    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-qjbkj"]
Jul 28 17:59:50.289440 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:50.289414    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="660e7892-5897-409d-8080-2fb9977fdf67" path="/var/lib/kubelet/pods/660e7892-5897-409d-8080-2fb9977fdf67/volumes"
Jul 28 17:59:51.527304 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.527265    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:59:51.604557 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.604526    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"]
Jul 28 17:59:51.604848 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.604832    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="660e7892-5897-409d-8080-2fb9977fdf67" containerName="kserve-container"
Jul 28 17:59:51.604901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.604849    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="660e7892-5897-409d-8080-2fb9977fdf67" containerName="kserve-container"
Jul 28 17:59:51.604901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.604865    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="660e7892-5897-409d-8080-2fb9977fdf67" containerName="storage-initializer"
Jul 28 17:59:51.604901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.604871    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="660e7892-5897-409d-8080-2fb9977fdf67" containerName="storage-initializer"
Jul 28 17:59:51.604901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.604878    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="660e7892-5897-409d-8080-2fb9977fdf67" containerName="kube-rbac-proxy"
Jul 28 17:59:51.604901 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.604884    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="660e7892-5897-409d-8080-2fb9977fdf67" containerName="kube-rbac-proxy"
Jul 28 17:59:51.605102 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.604930    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="660e7892-5897-409d-8080-2fb9977fdf67" containerName="kube-rbac-proxy"
Jul 28 17:59:51.605102 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.604953    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="660e7892-5897-409d-8080-2fb9977fdf67" containerName="kserve-container"
Jul 28 17:59:51.609230 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.609209    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"
Jul 28 17:59:51.611979 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.611956    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Jul 28 17:59:51.612087 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.612014    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 17:59:51.612164 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.612109    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:59:51.612226 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.612199    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:59:51.612287 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.612270    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:59:51.612479 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.612460    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Jul 28 17:59:51.613117 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.613101    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:59:51.622930 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.622908    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"]
Jul 28 17:59:51.759502 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.759471    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9daf85d7-614f-443d-8a84-6ffe73dd56f0-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-429j2\" (UID: \"9daf85d7-614f-443d-8a84-6ffe73dd56f0\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"
Jul 28 17:59:51.759704 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.759510    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7grxc\" (UniqueName: \"kubernetes.io/projected/9daf85d7-614f-443d-8a84-6ffe73dd56f0-kube-api-access-7grxc\") pod \"isvc-pmml-runtime-predictor-885fc8777-429j2\" (UID: \"9daf85d7-614f-443d-8a84-6ffe73dd56f0\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"
Jul 28 17:59:51.759704 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.759539    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9daf85d7-614f-443d-8a84-6ffe73dd56f0-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-429j2\" (UID: \"9daf85d7-614f-443d-8a84-6ffe73dd56f0\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"
Jul 28 17:59:51.759704 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.759591    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9daf85d7-614f-443d-8a84-6ffe73dd56f0-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-429j2\" (UID: \"9daf85d7-614f-443d-8a84-6ffe73dd56f0\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"
Jul 28 17:59:51.759704 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.759657    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9daf85d7-614f-443d-8a84-6ffe73dd56f0-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-429j2\" (UID: \"9daf85d7-614f-443d-8a84-6ffe73dd56f0\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"
Jul 28 17:59:51.860525 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.860447    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9daf85d7-614f-443d-8a84-6ffe73dd56f0-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-429j2\" (UID: \"9daf85d7-614f-443d-8a84-6ffe73dd56f0\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"
Jul 28 17:59:51.860525 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.860484    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7grxc\" (UniqueName: \"kubernetes.io/projected/9daf85d7-614f-443d-8a84-6ffe73dd56f0-kube-api-access-7grxc\") pod \"isvc-pmml-runtime-predictor-885fc8777-429j2\" (UID: \"9daf85d7-614f-443d-8a84-6ffe73dd56f0\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"
Jul 28 17:59:51.860525 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.860512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9daf85d7-614f-443d-8a84-6ffe73dd56f0-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-429j2\" (UID: \"9daf85d7-614f-443d-8a84-6ffe73dd56f0\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"
Jul 28 17:59:51.860750 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.860544    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9daf85d7-614f-443d-8a84-6ffe73dd56f0-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-429j2\" (UID: \"9daf85d7-614f-443d-8a84-6ffe73dd56f0\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"
Jul 28 17:59:51.860750 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.860603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9daf85d7-614f-443d-8a84-6ffe73dd56f0-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-429j2\" (UID: \"9daf85d7-614f-443d-8a84-6ffe73dd56f0\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"
Jul 28 17:59:51.860750 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:59:51.860691    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-runtime-predictor-serving-cert: secret "isvc-pmml-runtime-predictor-serving-cert" not found
Jul 28 17:59:51.860901 ip-10-0-138-231 kubenswrapper[2581]: E0728 17:59:51.860752    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9daf85d7-614f-443d-8a84-6ffe73dd56f0-proxy-tls podName:9daf85d7-614f-443d-8a84-6ffe73dd56f0 nodeName:}" failed. No retries permitted until 2026-07-28 17:59:52.3607322 +0000 UTC m=+3040.635348616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9daf85d7-614f-443d-8a84-6ffe73dd56f0-proxy-tls") pod "isvc-pmml-runtime-predictor-885fc8777-429j2" (UID: "9daf85d7-614f-443d-8a84-6ffe73dd56f0") : secret "isvc-pmml-runtime-predictor-serving-cert" not found
Jul 28 17:59:51.861050 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.861024    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9daf85d7-614f-443d-8a84-6ffe73dd56f0-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-429j2\" (UID: \"9daf85d7-614f-443d-8a84-6ffe73dd56f0\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"
Jul 28 17:59:51.861261 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.861244    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9daf85d7-614f-443d-8a84-6ffe73dd56f0-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-429j2\" (UID: \"9daf85d7-614f-443d-8a84-6ffe73dd56f0\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"
Jul 28 17:59:51.861381 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.861361    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9daf85d7-614f-443d-8a84-6ffe73dd56f0-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-429j2\" (UID: \"9daf85d7-614f-443d-8a84-6ffe73dd56f0\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"
Jul 28 17:59:51.881364 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.881336    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7grxc\" (UniqueName: \"kubernetes.io/projected/9daf85d7-614f-443d-8a84-6ffe73dd56f0-kube-api-access-7grxc\") pod \"isvc-pmml-runtime-predictor-885fc8777-429j2\" (UID: \"9daf85d7-614f-443d-8a84-6ffe73dd56f0\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"
Jul 28 17:59:51.970161 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:51.970124    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 17:59:52.364054 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:52.364020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9daf85d7-614f-443d-8a84-6ffe73dd56f0-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-429j2\" (UID: \"9daf85d7-614f-443d-8a84-6ffe73dd56f0\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"
Jul 28 17:59:52.366330 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:52.366301    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9daf85d7-614f-443d-8a84-6ffe73dd56f0-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-429j2\" (UID: \"9daf85d7-614f-443d-8a84-6ffe73dd56f0\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"
Jul 28 17:59:52.519349 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:52.519316    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"
Jul 28 17:59:52.647360 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:52.647337    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"]
Jul 28 17:59:52.649433 ip-10-0-138-231 kubenswrapper[2581]: W0728 17:59:52.649401    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9daf85d7_614f_443d_8a84_6ffe73dd56f0.slice/crio-4302cf0bbda1204cb3a77cd7f4f390cc9c23d2bf491821af6c3331fb96e6af05 WatchSource:0}: Error finding container 4302cf0bbda1204cb3a77cd7f4f390cc9c23d2bf491821af6c3331fb96e6af05: Status 404 returned error can't find the container with id 4302cf0bbda1204cb3a77cd7f4f390cc9c23d2bf491821af6c3331fb96e6af05
Jul 28 17:59:53.236957 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:53.236908    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2" event={"ID":"9daf85d7-614f-443d-8a84-6ffe73dd56f0","Type":"ContainerStarted","Data":"6873ea7f1bd47a2445d4aa7a0294a8537c01f5014533d451b56aa2c7f88f6d67"}
Jul 28 17:59:53.237134 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:53.237021    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2" event={"ID":"9daf85d7-614f-443d-8a84-6ffe73dd56f0","Type":"ContainerStarted","Data":"4302cf0bbda1204cb3a77cd7f4f390cc9c23d2bf491821af6c3331fb96e6af05"}
Jul 28 17:59:57.251922 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:57.251879    2581 generic.go:356] "Generic (PLEG): container finished" podID="9daf85d7-614f-443d-8a84-6ffe73dd56f0" containerID="6873ea7f1bd47a2445d4aa7a0294a8537c01f5014533d451b56aa2c7f88f6d67" exitCode=0
Jul 28 17:59:57.251922 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:57.251921    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2" event={"ID":"9daf85d7-614f-443d-8a84-6ffe73dd56f0","Type":"ContainerDied","Data":"6873ea7f1bd47a2445d4aa7a0294a8537c01f5014533d451b56aa2c7f88f6d67"}
Jul 28 17:59:58.257416 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:58.257380    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2" event={"ID":"9daf85d7-614f-443d-8a84-6ffe73dd56f0","Type":"ContainerStarted","Data":"612c6b8e26939b9c6de9b8ce4413aceafc7d548b867e25a7dfd5180247f8f350"}
Jul 28 17:59:58.257857 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:58.257444    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2" event={"ID":"9daf85d7-614f-443d-8a84-6ffe73dd56f0","Type":"ContainerStarted","Data":"d45836d376d428a23d7bcceb975dc9dc0153e71ff5b65af83bf71d39732da5c6"}
Jul 28 17:59:58.257857 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:58.257694    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"
Jul 28 17:59:58.257857 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:58.257840    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"
Jul 28 17:59:58.259371 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:58.259342    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2" podUID="9daf85d7-614f-443d-8a84-6ffe73dd56f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 17:59:58.295606 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:58.295565    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2" podStartSLOduration=3.500615174 podStartE2EDuration="7.295550456s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.794935282s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:59:58.295350388 +0000 UTC m=+3046.569966821" watchObservedRunningTime="2026-07-28 17:59:58.295550456 +0000 UTC m=+3046.570166887"
Jul 28 17:59:59.261397 ip-10-0-138-231 kubenswrapper[2581]: I0728 17:59:59.261356    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2" podUID="9daf85d7-614f-443d-8a84-6ffe73dd56f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 18:00:04.266239 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:00:04.266207    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"
Jul 28 18:00:04.266637 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:00:04.266611    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2" podUID="9daf85d7-614f-443d-8a84-6ffe73dd56f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 18:00:14.267422 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:00:14.267379    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2" podUID="9daf85d7-614f-443d-8a84-6ffe73dd56f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 18:00:24.267456 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:00:24.267410    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2" podUID="9daf85d7-614f-443d-8a84-6ffe73dd56f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 18:00:34.267186 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:00:34.267148    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2" podUID="9daf85d7-614f-443d-8a84-6ffe73dd56f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 18:00:44.266905 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:00:44.266868    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2" podUID="9daf85d7-614f-443d-8a84-6ffe73dd56f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 18:00:54.267437 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:00:54.267397    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2" podUID="9daf85d7-614f-443d-8a84-6ffe73dd56f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 18:01:04.267560 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:04.267525    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2" podUID="9daf85d7-614f-443d-8a84-6ffe73dd56f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 18:01:14.267611 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:14.267578    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"
Jul 28 18:01:22.774306 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:22.774269    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"]
Jul 28 18:01:22.774862 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:22.774512    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2" podUID="9daf85d7-614f-443d-8a84-6ffe73dd56f0" containerName="kserve-container" containerID="cri-o://d45836d376d428a23d7bcceb975dc9dc0153e71ff5b65af83bf71d39732da5c6" gracePeriod=30
Jul 28 18:01:22.774862 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:22.774585    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2" podUID="9daf85d7-614f-443d-8a84-6ffe73dd56f0" containerName="kube-rbac-proxy" containerID="cri-o://612c6b8e26939b9c6de9b8ce4413aceafc7d548b867e25a7dfd5180247f8f350" gracePeriod=30
Jul 28 18:01:22.812708 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:22.812675    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:01:23.550883 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:23.550845    2581 generic.go:356] "Generic (PLEG): container finished" podID="9daf85d7-614f-443d-8a84-6ffe73dd56f0" containerID="612c6b8e26939b9c6de9b8ce4413aceafc7d548b867e25a7dfd5180247f8f350" exitCode=2
Jul 28 18:01:23.550883 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:23.550884    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2" event={"ID":"9daf85d7-614f-443d-8a84-6ffe73dd56f0","Type":"ContainerDied","Data":"612c6b8e26939b9c6de9b8ce4413aceafc7d548b867e25a7dfd5180247f8f350"}
Jul 28 18:01:24.262131 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:24.262087    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2" podUID="9daf85d7-614f-443d-8a84-6ffe73dd56f0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Jul 28 18:01:24.267447 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:24.267425    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2" podUID="9daf85d7-614f-443d-8a84-6ffe73dd56f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 28 18:01:26.011219 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.011195    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"
Jul 28 18:01:26.141566 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.141489    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9daf85d7-614f-443d-8a84-6ffe73dd56f0-cabundle-cert\") pod \"9daf85d7-614f-443d-8a84-6ffe73dd56f0\" (UID: \"9daf85d7-614f-443d-8a84-6ffe73dd56f0\") "
Jul 28 18:01:26.141566 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.141540    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7grxc\" (UniqueName: \"kubernetes.io/projected/9daf85d7-614f-443d-8a84-6ffe73dd56f0-kube-api-access-7grxc\") pod \"9daf85d7-614f-443d-8a84-6ffe73dd56f0\" (UID: \"9daf85d7-614f-443d-8a84-6ffe73dd56f0\") "
Jul 28 18:01:26.141794 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.141586    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9daf85d7-614f-443d-8a84-6ffe73dd56f0-proxy-tls\") pod \"9daf85d7-614f-443d-8a84-6ffe73dd56f0\" (UID: \"9daf85d7-614f-443d-8a84-6ffe73dd56f0\") "
Jul 28 18:01:26.141794 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.141618    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9daf85d7-614f-443d-8a84-6ffe73dd56f0-kube-rbac-proxy-sar-config\") pod \"9daf85d7-614f-443d-8a84-6ffe73dd56f0\" (UID: \"9daf85d7-614f-443d-8a84-6ffe73dd56f0\") "
Jul 28 18:01:26.141794 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.141654    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9daf85d7-614f-443d-8a84-6ffe73dd56f0-kserve-provision-location\") pod \"9daf85d7-614f-443d-8a84-6ffe73dd56f0\" (UID: \"9daf85d7-614f-443d-8a84-6ffe73dd56f0\") "
Jul 28 18:01:26.141979 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.141907    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9daf85d7-614f-443d-8a84-6ffe73dd56f0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9daf85d7-614f-443d-8a84-6ffe73dd56f0" (UID: "9daf85d7-614f-443d-8a84-6ffe73dd56f0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:01:26.142046 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.141996    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9daf85d7-614f-443d-8a84-6ffe73dd56f0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9daf85d7-614f-443d-8a84-6ffe73dd56f0" (UID: "9daf85d7-614f-443d-8a84-6ffe73dd56f0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:01:26.142087 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.142041    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9daf85d7-614f-443d-8a84-6ffe73dd56f0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9daf85d7-614f-443d-8a84-6ffe73dd56f0" (UID: "9daf85d7-614f-443d-8a84-6ffe73dd56f0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:01:26.143719 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.143700    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9daf85d7-614f-443d-8a84-6ffe73dd56f0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9daf85d7-614f-443d-8a84-6ffe73dd56f0" (UID: "9daf85d7-614f-443d-8a84-6ffe73dd56f0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:01:26.143786 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.143777    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9daf85d7-614f-443d-8a84-6ffe73dd56f0-kube-api-access-7grxc" (OuterVolumeSpecName: "kube-api-access-7grxc") pod "9daf85d7-614f-443d-8a84-6ffe73dd56f0" (UID: "9daf85d7-614f-443d-8a84-6ffe73dd56f0"). InnerVolumeSpecName "kube-api-access-7grxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:01:26.242823 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.242799    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7grxc\" (UniqueName: \"kubernetes.io/projected/9daf85d7-614f-443d-8a84-6ffe73dd56f0-kube-api-access-7grxc\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:01:26.242823 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.242819    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9daf85d7-614f-443d-8a84-6ffe73dd56f0-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:01:26.242823 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.242828    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9daf85d7-614f-443d-8a84-6ffe73dd56f0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:01:26.243042 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.242837    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9daf85d7-614f-443d-8a84-6ffe73dd56f0-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:01:26.243042 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.242846    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9daf85d7-614f-443d-8a84-6ffe73dd56f0-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:01:26.562557 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.562525    2581 generic.go:356] "Generic (PLEG): container finished" podID="9daf85d7-614f-443d-8a84-6ffe73dd56f0" containerID="d45836d376d428a23d7bcceb975dc9dc0153e71ff5b65af83bf71d39732da5c6" exitCode=0
Jul 28 18:01:26.562732 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.562584    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"
Jul 28 18:01:26.562732 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.562605    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2" event={"ID":"9daf85d7-614f-443d-8a84-6ffe73dd56f0","Type":"ContainerDied","Data":"d45836d376d428a23d7bcceb975dc9dc0153e71ff5b65af83bf71d39732da5c6"}
Jul 28 18:01:26.562732 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.562660    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2" event={"ID":"9daf85d7-614f-443d-8a84-6ffe73dd56f0","Type":"ContainerDied","Data":"4302cf0bbda1204cb3a77cd7f4f390cc9c23d2bf491821af6c3331fb96e6af05"}
Jul 28 18:01:26.562732 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.562701    2581 scope.go:117] "RemoveContainer" containerID="612c6b8e26939b9c6de9b8ce4413aceafc7d548b867e25a7dfd5180247f8f350"
Jul 28 18:01:26.570235 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.570218    2581 scope.go:117] "RemoveContainer" containerID="d45836d376d428a23d7bcceb975dc9dc0153e71ff5b65af83bf71d39732da5c6"
Jul 28 18:01:26.583512 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.583478    2581 scope.go:117] "RemoveContainer" containerID="6873ea7f1bd47a2445d4aa7a0294a8537c01f5014533d451b56aa2c7f88f6d67"
Jul 28 18:01:26.589901 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.589877    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"]
Jul 28 18:01:26.591134 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.591115    2581 scope.go:117] "RemoveContainer" containerID="612c6b8e26939b9c6de9b8ce4413aceafc7d548b867e25a7dfd5180247f8f350"
Jul 28 18:01:26.591386 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:01:26.591357    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"612c6b8e26939b9c6de9b8ce4413aceafc7d548b867e25a7dfd5180247f8f350\": container with ID starting with 612c6b8e26939b9c6de9b8ce4413aceafc7d548b867e25a7dfd5180247f8f350 not found: ID does not exist" containerID="612c6b8e26939b9c6de9b8ce4413aceafc7d548b867e25a7dfd5180247f8f350"
Jul 28 18:01:26.591446 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.591384    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"612c6b8e26939b9c6de9b8ce4413aceafc7d548b867e25a7dfd5180247f8f350"} err="failed to get container status \"612c6b8e26939b9c6de9b8ce4413aceafc7d548b867e25a7dfd5180247f8f350\": rpc error: code = NotFound desc = could not find container \"612c6b8e26939b9c6de9b8ce4413aceafc7d548b867e25a7dfd5180247f8f350\": container with ID starting with 612c6b8e26939b9c6de9b8ce4413aceafc7d548b867e25a7dfd5180247f8f350 not found: ID does not exist"
Jul 28 18:01:26.591446 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.591403    2581 scope.go:117] "RemoveContainer" containerID="d45836d376d428a23d7bcceb975dc9dc0153e71ff5b65af83bf71d39732da5c6"
Jul 28 18:01:26.591632 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:01:26.591613    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d45836d376d428a23d7bcceb975dc9dc0153e71ff5b65af83bf71d39732da5c6\": container with ID starting with d45836d376d428a23d7bcceb975dc9dc0153e71ff5b65af83bf71d39732da5c6 not found: ID does not exist" containerID="d45836d376d428a23d7bcceb975dc9dc0153e71ff5b65af83bf71d39732da5c6"
Jul 28 18:01:26.591695 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.591642    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d45836d376d428a23d7bcceb975dc9dc0153e71ff5b65af83bf71d39732da5c6"} err="failed to get container status \"d45836d376d428a23d7bcceb975dc9dc0153e71ff5b65af83bf71d39732da5c6\": rpc error: code = NotFound desc = could not find container \"d45836d376d428a23d7bcceb975dc9dc0153e71ff5b65af83bf71d39732da5c6\": container with ID starting with d45836d376d428a23d7bcceb975dc9dc0153e71ff5b65af83bf71d39732da5c6 not found: ID does not exist"
Jul 28 18:01:26.591695 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.591666    2581 scope.go:117] "RemoveContainer" containerID="6873ea7f1bd47a2445d4aa7a0294a8537c01f5014533d451b56aa2c7f88f6d67"
Jul 28 18:01:26.591893 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:01:26.591877    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6873ea7f1bd47a2445d4aa7a0294a8537c01f5014533d451b56aa2c7f88f6d67\": container with ID starting with 6873ea7f1bd47a2445d4aa7a0294a8537c01f5014533d451b56aa2c7f88f6d67 not found: ID does not exist" containerID="6873ea7f1bd47a2445d4aa7a0294a8537c01f5014533d451b56aa2c7f88f6d67"
Jul 28 18:01:26.591974 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.591901    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6873ea7f1bd47a2445d4aa7a0294a8537c01f5014533d451b56aa2c7f88f6d67"} err="failed to get container status \"6873ea7f1bd47a2445d4aa7a0294a8537c01f5014533d451b56aa2c7f88f6d67\": rpc error: code = NotFound desc = could not find container \"6873ea7f1bd47a2445d4aa7a0294a8537c01f5014533d451b56aa2c7f88f6d67\": container with ID starting with 6873ea7f1bd47a2445d4aa7a0294a8537c01f5014533d451b56aa2c7f88f6d67 not found: ID does not exist"
Jul 28 18:01:26.594514 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:26.594495    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-429j2"]
Jul 28 18:01:28.023277 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.023238    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:01:28.085221 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.085191    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"]
Jul 28 18:01:28.085471 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.085459    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9daf85d7-614f-443d-8a84-6ffe73dd56f0" containerName="storage-initializer"
Jul 28 18:01:28.085521 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.085472    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9daf85d7-614f-443d-8a84-6ffe73dd56f0" containerName="storage-initializer"
Jul 28 18:01:28.085521 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.085481    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9daf85d7-614f-443d-8a84-6ffe73dd56f0" containerName="kserve-container"
Jul 28 18:01:28.085521 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.085487    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9daf85d7-614f-443d-8a84-6ffe73dd56f0" containerName="kserve-container"
Jul 28 18:01:28.085521 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.085500    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9daf85d7-614f-443d-8a84-6ffe73dd56f0" containerName="kube-rbac-proxy"
Jul 28 18:01:28.085521 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.085506    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9daf85d7-614f-443d-8a84-6ffe73dd56f0" containerName="kube-rbac-proxy"
Jul 28 18:01:28.085669 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.085544    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9daf85d7-614f-443d-8a84-6ffe73dd56f0" containerName="kube-rbac-proxy"
Jul 28 18:01:28.085669 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.085555    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9daf85d7-614f-443d-8a84-6ffe73dd56f0" containerName="kserve-container"
Jul 28 18:01:28.089837 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.089821    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"
Jul 28 18:01:28.093652 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.093629    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 18:01:28.094262 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.094247    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:01:28.094518 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.094490    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:01:28.094518 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.094513    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 28 18:01:28.094691 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.094515    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:01:28.094790 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.094771    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Jul 28 18:01:28.094889 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.094872    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:01:28.111086 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.111063    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"]
Jul 28 18:01:28.256399 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.256369    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69cd0ad2-dea6-4368-ba57-62d70613d679-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl\" (UID: \"69cd0ad2-dea6-4368-ba57-62d70613d679\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"
Jul 28 18:01:28.256548 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.256422    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69cd0ad2-dea6-4368-ba57-62d70613d679-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl\" (UID: \"69cd0ad2-dea6-4368-ba57-62d70613d679\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"
Jul 28 18:01:28.256548 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.256452    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69cd0ad2-dea6-4368-ba57-62d70613d679-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl\" (UID: \"69cd0ad2-dea6-4368-ba57-62d70613d679\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"
Jul 28 18:01:28.256548 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.256477    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69cd0ad2-dea6-4368-ba57-62d70613d679-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl\" (UID: \"69cd0ad2-dea6-4368-ba57-62d70613d679\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"
Jul 28 18:01:28.256548 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.256503    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hwhm\" (UniqueName: \"kubernetes.io/projected/69cd0ad2-dea6-4368-ba57-62d70613d679-kube-api-access-8hwhm\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl\" (UID: \"69cd0ad2-dea6-4368-ba57-62d70613d679\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"
Jul 28 18:01:28.290998 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.290918    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9daf85d7-614f-443d-8a84-6ffe73dd56f0" path="/var/lib/kubelet/pods/9daf85d7-614f-443d-8a84-6ffe73dd56f0/volumes"
Jul 28 18:01:28.357146 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.357115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69cd0ad2-dea6-4368-ba57-62d70613d679-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl\" (UID: \"69cd0ad2-dea6-4368-ba57-62d70613d679\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"
Jul 28 18:01:28.357146 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.357145    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69cd0ad2-dea6-4368-ba57-62d70613d679-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl\" (UID: \"69cd0ad2-dea6-4368-ba57-62d70613d679\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"
Jul 28 18:01:28.357304 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.357171    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69cd0ad2-dea6-4368-ba57-62d70613d679-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl\" (UID: \"69cd0ad2-dea6-4368-ba57-62d70613d679\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"
Jul 28 18:01:28.357304 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.357203    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hwhm\" (UniqueName: \"kubernetes.io/projected/69cd0ad2-dea6-4368-ba57-62d70613d679-kube-api-access-8hwhm\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl\" (UID: \"69cd0ad2-dea6-4368-ba57-62d70613d679\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"
Jul 28 18:01:28.357304 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.357235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69cd0ad2-dea6-4368-ba57-62d70613d679-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl\" (UID: \"69cd0ad2-dea6-4368-ba57-62d70613d679\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"
Jul 28 18:01:28.357569 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.357550    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69cd0ad2-dea6-4368-ba57-62d70613d679-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl\" (UID: \"69cd0ad2-dea6-4368-ba57-62d70613d679\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"
Jul 28 18:01:28.357862 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.357842    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69cd0ad2-dea6-4368-ba57-62d70613d679-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl\" (UID: \"69cd0ad2-dea6-4368-ba57-62d70613d679\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"
Jul 28 18:01:28.357931 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.357870    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69cd0ad2-dea6-4368-ba57-62d70613d679-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl\" (UID: \"69cd0ad2-dea6-4368-ba57-62d70613d679\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"
Jul 28 18:01:28.359565 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.359545    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69cd0ad2-dea6-4368-ba57-62d70613d679-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl\" (UID: \"69cd0ad2-dea6-4368-ba57-62d70613d679\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"
Jul 28 18:01:28.371632 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.371607    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hwhm\" (UniqueName: \"kubernetes.io/projected/69cd0ad2-dea6-4368-ba57-62d70613d679-kube-api-access-8hwhm\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl\" (UID: \"69cd0ad2-dea6-4368-ba57-62d70613d679\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"
Jul 28 18:01:28.399491 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.399470    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"
Jul 28 18:01:28.425502 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.425476    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:01:28.535531 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.535499    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"]
Jul 28 18:01:28.538538 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:01:28.538515    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69cd0ad2_dea6_4368_ba57_62d70613d679.slice/crio-230ff32ccd473d67d7711a1480591b1fbbb8928d171058cd79ff055d6d85971d WatchSource:0}: Error finding container 230ff32ccd473d67d7711a1480591b1fbbb8928d171058cd79ff055d6d85971d: Status 404 returned error can't find the container with id 230ff32ccd473d67d7711a1480591b1fbbb8928d171058cd79ff055d6d85971d
Jul 28 18:01:28.571036 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:28.571010    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl" event={"ID":"69cd0ad2-dea6-4368-ba57-62d70613d679","Type":"ContainerStarted","Data":"230ff32ccd473d67d7711a1480591b1fbbb8928d171058cd79ff055d6d85971d"}
Jul 28 18:01:29.576249 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:29.576215    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl" event={"ID":"69cd0ad2-dea6-4368-ba57-62d70613d679","Type":"ContainerStarted","Data":"0863d2aa064cd28a3bf48ebf84bea52b186ddbfba0e7c77c97c497419da362da"}
Jul 28 18:01:32.588630 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:32.588538    2581 generic.go:356] "Generic (PLEG): container finished" podID="69cd0ad2-dea6-4368-ba57-62d70613d679" containerID="0863d2aa064cd28a3bf48ebf84bea52b186ddbfba0e7c77c97c497419da362da" exitCode=0
Jul 28 18:01:32.588630 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:32.588592    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl" event={"ID":"69cd0ad2-dea6-4368-ba57-62d70613d679","Type":"ContainerDied","Data":"0863d2aa064cd28a3bf48ebf84bea52b186ddbfba0e7c77c97c497419da362da"}
Jul 28 18:01:33.593658 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:33.593619    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl" event={"ID":"69cd0ad2-dea6-4368-ba57-62d70613d679","Type":"ContainerStarted","Data":"4925777e98cfa88f9fc83ccccefc1eaefa3e8326fba07f6b75dab66974b97516"}
Jul 28 18:01:33.594068 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:33.593685    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl" event={"ID":"69cd0ad2-dea6-4368-ba57-62d70613d679","Type":"ContainerStarted","Data":"63411b3349566e532e2d59f7136bfe62e45d08e38ae8652a0409a4b75609e7f8"}
Jul 28 18:01:33.594068 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:33.593870    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"
Jul 28 18:01:33.616586 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:33.616545    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl" podStartSLOduration=1.918342642 podStartE2EDuration="5.616531809s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.698189167s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:33.616483387 +0000 UTC m=+3141.891099820" watchObservedRunningTime="2026-07-28 18:01:33.616531809 +0000 UTC m=+3141.891148240"
Jul 28 18:01:34.597559 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:34.597527    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"
Jul 28 18:01:34.598527 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:34.598498    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl" podUID="69cd0ad2-dea6-4368-ba57-62d70613d679" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 18:01:35.601017 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:35.600978    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl" podUID="69cd0ad2-dea6-4368-ba57-62d70613d679" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 18:01:40.605442 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:40.605407    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"
Jul 28 18:01:40.605892 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:40.605865    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl" podUID="69cd0ad2-dea6-4368-ba57-62d70613d679" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 18:01:50.605823 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:01:50.605785    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl" podUID="69cd0ad2-dea6-4368-ba57-62d70613d679" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 18:02:00.606874 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:02:00.606832    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl" podUID="69cd0ad2-dea6-4368-ba57-62d70613d679" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 18:02:10.606480 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:02:10.606435    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl" podUID="69cd0ad2-dea6-4368-ba57-62d70613d679" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 18:02:20.606325 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:02:20.606282    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl" podUID="69cd0ad2-dea6-4368-ba57-62d70613d679" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 18:02:30.605829 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:02:30.605786    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl" podUID="69cd0ad2-dea6-4368-ba57-62d70613d679" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 18:02:40.606079 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:02:40.606034    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl" podUID="69cd0ad2-dea6-4368-ba57-62d70613d679" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 18:02:50.607115 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:02:50.607081    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"
Jul 28 18:02:59.068628 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:02:59.068585    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"]
Jul 28 18:02:59.069141 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:02:59.068928    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl" podUID="69cd0ad2-dea6-4368-ba57-62d70613d679" containerName="kserve-container" containerID="cri-o://63411b3349566e532e2d59f7136bfe62e45d08e38ae8652a0409a4b75609e7f8" gracePeriod=30
Jul 28 18:02:59.069141 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:02:59.069023    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl" podUID="69cd0ad2-dea6-4368-ba57-62d70613d679" containerName="kube-rbac-proxy" containerID="cri-o://4925777e98cfa88f9fc83ccccefc1eaefa3e8326fba07f6b75dab66974b97516" gracePeriod=30
Jul 28 18:02:59.605669 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:02:59.605626    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:02:59.897932 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:02:59.897832    2581 generic.go:356] "Generic (PLEG): container finished" podID="69cd0ad2-dea6-4368-ba57-62d70613d679" containerID="4925777e98cfa88f9fc83ccccefc1eaefa3e8326fba07f6b75dab66974b97516" exitCode=2
Jul 28 18:02:59.898107 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:02:59.897908    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl" event={"ID":"69cd0ad2-dea6-4368-ba57-62d70613d679","Type":"ContainerDied","Data":"4925777e98cfa88f9fc83ccccefc1eaefa3e8326fba07f6b75dab66974b97516"}
Jul 28 18:03:00.601550 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:00.601499    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl" podUID="69cd0ad2-dea6-4368-ba57-62d70613d679" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Jul 28 18:03:00.606825 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:00.606801    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl" podUID="69cd0ad2-dea6-4368-ba57-62d70613d679" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 18:03:02.713563 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.713534    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"
Jul 28 18:03:02.857863 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.857776    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hwhm\" (UniqueName: \"kubernetes.io/projected/69cd0ad2-dea6-4368-ba57-62d70613d679-kube-api-access-8hwhm\") pod \"69cd0ad2-dea6-4368-ba57-62d70613d679\" (UID: \"69cd0ad2-dea6-4368-ba57-62d70613d679\") "
Jul 28 18:03:02.857863 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.857822    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69cd0ad2-dea6-4368-ba57-62d70613d679-kube-rbac-proxy-sar-config\") pod \"69cd0ad2-dea6-4368-ba57-62d70613d679\" (UID: \"69cd0ad2-dea6-4368-ba57-62d70613d679\") "
Jul 28 18:03:02.858108 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.857879    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69cd0ad2-dea6-4368-ba57-62d70613d679-cabundle-cert\") pod \"69cd0ad2-dea6-4368-ba57-62d70613d679\" (UID: \"69cd0ad2-dea6-4368-ba57-62d70613d679\") "
Jul 28 18:03:02.858108 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.857900    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69cd0ad2-dea6-4368-ba57-62d70613d679-kserve-provision-location\") pod \"69cd0ad2-dea6-4368-ba57-62d70613d679\" (UID: \"69cd0ad2-dea6-4368-ba57-62d70613d679\") "
Jul 28 18:03:02.858108 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.857932    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69cd0ad2-dea6-4368-ba57-62d70613d679-proxy-tls\") pod \"69cd0ad2-dea6-4368-ba57-62d70613d679\" (UID: \"69cd0ad2-dea6-4368-ba57-62d70613d679\") "
Jul 28 18:03:02.858309 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.858279    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69cd0ad2-dea6-4368-ba57-62d70613d679-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "69cd0ad2-dea6-4368-ba57-62d70613d679" (UID: "69cd0ad2-dea6-4368-ba57-62d70613d679"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:03:02.858309 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.858298    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69cd0ad2-dea6-4368-ba57-62d70613d679-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "69cd0ad2-dea6-4368-ba57-62d70613d679" (UID: "69cd0ad2-dea6-4368-ba57-62d70613d679"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:03:02.858469 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.858296    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69cd0ad2-dea6-4368-ba57-62d70613d679-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "69cd0ad2-dea6-4368-ba57-62d70613d679" (UID: "69cd0ad2-dea6-4368-ba57-62d70613d679"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:03:02.860182 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.860156    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69cd0ad2-dea6-4368-ba57-62d70613d679-kube-api-access-8hwhm" (OuterVolumeSpecName: "kube-api-access-8hwhm") pod "69cd0ad2-dea6-4368-ba57-62d70613d679" (UID: "69cd0ad2-dea6-4368-ba57-62d70613d679"). InnerVolumeSpecName "kube-api-access-8hwhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:03:02.860267 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.860154    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69cd0ad2-dea6-4368-ba57-62d70613d679-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "69cd0ad2-dea6-4368-ba57-62d70613d679" (UID: "69cd0ad2-dea6-4368-ba57-62d70613d679"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:03:02.910831 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.910794    2581 generic.go:356] "Generic (PLEG): container finished" podID="69cd0ad2-dea6-4368-ba57-62d70613d679" containerID="63411b3349566e532e2d59f7136bfe62e45d08e38ae8652a0409a4b75609e7f8" exitCode=0
Jul 28 18:03:02.910992 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.910839    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl" event={"ID":"69cd0ad2-dea6-4368-ba57-62d70613d679","Type":"ContainerDied","Data":"63411b3349566e532e2d59f7136bfe62e45d08e38ae8652a0409a4b75609e7f8"}
Jul 28 18:03:02.910992 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.910848    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"
Jul 28 18:03:02.910992 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.910891    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl" event={"ID":"69cd0ad2-dea6-4368-ba57-62d70613d679","Type":"ContainerDied","Data":"230ff32ccd473d67d7711a1480591b1fbbb8928d171058cd79ff055d6d85971d"}
Jul 28 18:03:02.910992 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.910917    2581 scope.go:117] "RemoveContainer" containerID="4925777e98cfa88f9fc83ccccefc1eaefa3e8326fba07f6b75dab66974b97516"
Jul 28 18:03:02.918956 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.918915    2581 scope.go:117] "RemoveContainer" containerID="63411b3349566e532e2d59f7136bfe62e45d08e38ae8652a0409a4b75609e7f8"
Jul 28 18:03:02.926161 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.926143    2581 scope.go:117] "RemoveContainer" containerID="0863d2aa064cd28a3bf48ebf84bea52b186ddbfba0e7c77c97c497419da362da"
Jul 28 18:03:02.933259 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.933241    2581 scope.go:117] "RemoveContainer" containerID="4925777e98cfa88f9fc83ccccefc1eaefa3e8326fba07f6b75dab66974b97516"
Jul 28 18:03:02.933551 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:03:02.933532    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4925777e98cfa88f9fc83ccccefc1eaefa3e8326fba07f6b75dab66974b97516\": container with ID starting with 4925777e98cfa88f9fc83ccccefc1eaefa3e8326fba07f6b75dab66974b97516 not found: ID does not exist" containerID="4925777e98cfa88f9fc83ccccefc1eaefa3e8326fba07f6b75dab66974b97516"
Jul 28 18:03:02.933639 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.933563    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4925777e98cfa88f9fc83ccccefc1eaefa3e8326fba07f6b75dab66974b97516"} err="failed to get container status \"4925777e98cfa88f9fc83ccccefc1eaefa3e8326fba07f6b75dab66974b97516\": rpc error: code = NotFound desc = could not find container \"4925777e98cfa88f9fc83ccccefc1eaefa3e8326fba07f6b75dab66974b97516\": container with ID starting with 4925777e98cfa88f9fc83ccccefc1eaefa3e8326fba07f6b75dab66974b97516 not found: ID does not exist"
Jul 28 18:03:02.933639 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.933591    2581 scope.go:117] "RemoveContainer" containerID="63411b3349566e532e2d59f7136bfe62e45d08e38ae8652a0409a4b75609e7f8"
Jul 28 18:03:02.933860 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:03:02.933839    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63411b3349566e532e2d59f7136bfe62e45d08e38ae8652a0409a4b75609e7f8\": container with ID starting with 63411b3349566e532e2d59f7136bfe62e45d08e38ae8652a0409a4b75609e7f8 not found: ID does not exist" containerID="63411b3349566e532e2d59f7136bfe62e45d08e38ae8652a0409a4b75609e7f8"
Jul 28 18:03:02.933937 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.933868    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63411b3349566e532e2d59f7136bfe62e45d08e38ae8652a0409a4b75609e7f8"} err="failed to get container status \"63411b3349566e532e2d59f7136bfe62e45d08e38ae8652a0409a4b75609e7f8\": rpc error: code = NotFound desc = could not find container \"63411b3349566e532e2d59f7136bfe62e45d08e38ae8652a0409a4b75609e7f8\": container with ID starting with 63411b3349566e532e2d59f7136bfe62e45d08e38ae8652a0409a4b75609e7f8 not found: ID does not exist"
Jul 28 18:03:02.933937 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.933904    2581 scope.go:117] "RemoveContainer" containerID="0863d2aa064cd28a3bf48ebf84bea52b186ddbfba0e7c77c97c497419da362da"
Jul 28 18:03:02.934055 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.933985    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"]
Jul 28 18:03:02.934177 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:03:02.934156    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0863d2aa064cd28a3bf48ebf84bea52b186ddbfba0e7c77c97c497419da362da\": container with ID starting with 0863d2aa064cd28a3bf48ebf84bea52b186ddbfba0e7c77c97c497419da362da not found: ID does not exist" containerID="0863d2aa064cd28a3bf48ebf84bea52b186ddbfba0e7c77c97c497419da362da"
Jul 28 18:03:02.934217 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.934183    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0863d2aa064cd28a3bf48ebf84bea52b186ddbfba0e7c77c97c497419da362da"} err="failed to get container status \"0863d2aa064cd28a3bf48ebf84bea52b186ddbfba0e7c77c97c497419da362da\": rpc error: code = NotFound desc = could not find container \"0863d2aa064cd28a3bf48ebf84bea52b186ddbfba0e7c77c97c497419da362da\": container with ID starting with 0863d2aa064cd28a3bf48ebf84bea52b186ddbfba0e7c77c97c497419da362da not found: ID does not exist"
Jul 28 18:03:02.938020 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.937997    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-htpsl"]
Jul 28 18:03:02.959442 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.959406    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69cd0ad2-dea6-4368-ba57-62d70613d679-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:03:02.959442 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.959437    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69cd0ad2-dea6-4368-ba57-62d70613d679-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:03:02.959442 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.959448    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69cd0ad2-dea6-4368-ba57-62d70613d679-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:03:02.959442 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.959457    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hwhm\" (UniqueName: \"kubernetes.io/projected/69cd0ad2-dea6-4368-ba57-62d70613d679-kube-api-access-8hwhm\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:03:02.959737 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:02.959466    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69cd0ad2-dea6-4368-ba57-62d70613d679-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:03:04.083672 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.083628    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:04.176467 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.176419    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"]
Jul 28 18:03:04.176682 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.176670    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69cd0ad2-dea6-4368-ba57-62d70613d679" containerName="storage-initializer"
Jul 28 18:03:04.176725 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.176684    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="69cd0ad2-dea6-4368-ba57-62d70613d679" containerName="storage-initializer"
Jul 28 18:03:04.176725 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.176697    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69cd0ad2-dea6-4368-ba57-62d70613d679" containerName="kserve-container"
Jul 28 18:03:04.176725 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.176704    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="69cd0ad2-dea6-4368-ba57-62d70613d679" containerName="kserve-container"
Jul 28 18:03:04.176725 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.176717    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69cd0ad2-dea6-4368-ba57-62d70613d679" containerName="kube-rbac-proxy"
Jul 28 18:03:04.176725 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.176722    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="69cd0ad2-dea6-4368-ba57-62d70613d679" containerName="kube-rbac-proxy"
Jul 28 18:03:04.176881 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.176765    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="69cd0ad2-dea6-4368-ba57-62d70613d679" containerName="kserve-container"
Jul 28 18:03:04.176881 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.176774    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="69cd0ad2-dea6-4368-ba57-62d70613d679" containerName="kube-rbac-proxy"
Jul 28 18:03:04.181135 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.181117    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"
Jul 28 18:03:04.184194 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.184171    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:03:04.184358 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.184332    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:03:04.185410 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.185387    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:03:04.185410 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.185400    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-e02d20-kube-rbac-proxy-sar-config\""
Jul 28 18:03:04.185592 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.185411    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:03:04.185592 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.185421    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 18:03:04.185592 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.185400    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:03:04.185592 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.185394    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-e02d20-predictor-serving-cert\""
Jul 28 18:03:04.191791 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.191768    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"]
Jul 28 18:03:04.269827 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.269787    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-kserve-provision-location\") pod \"isvc-primary-e02d20-predictor-5cc5bf6845-57s2f\" (UID: \"ab659d35-5ddd-4283-8bc8-e3b39f98d61c\") " pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"
Jul 28 18:03:04.270022 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.269838    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-cabundle-cert\") pod \"isvc-primary-e02d20-predictor-5cc5bf6845-57s2f\" (UID: \"ab659d35-5ddd-4283-8bc8-e3b39f98d61c\") " pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"
Jul 28 18:03:04.270022 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.269907    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-kube-rbac-proxy-sar-config\") pod \"isvc-primary-e02d20-predictor-5cc5bf6845-57s2f\" (UID: \"ab659d35-5ddd-4283-8bc8-e3b39f98d61c\") " pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"
Jul 28 18:03:04.270022 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.269989    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-proxy-tls\") pod \"isvc-primary-e02d20-predictor-5cc5bf6845-57s2f\" (UID: \"ab659d35-5ddd-4283-8bc8-e3b39f98d61c\") " pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"
Jul 28 18:03:04.270022 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.270020    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzw22\" (UniqueName: \"kubernetes.io/projected/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-kube-api-access-zzw22\") pod \"isvc-primary-e02d20-predictor-5cc5bf6845-57s2f\" (UID: \"ab659d35-5ddd-4283-8bc8-e3b39f98d61c\") " pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"
Jul 28 18:03:04.289321 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.289292    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69cd0ad2-dea6-4368-ba57-62d70613d679" path="/var/lib/kubelet/pods/69cd0ad2-dea6-4368-ba57-62d70613d679/volumes"
Jul 28 18:03:04.370985 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.370876    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-kserve-provision-location\") pod \"isvc-primary-e02d20-predictor-5cc5bf6845-57s2f\" (UID: \"ab659d35-5ddd-4283-8bc8-e3b39f98d61c\") " pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"
Jul 28 18:03:04.371094 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.370996    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-cabundle-cert\") pod \"isvc-primary-e02d20-predictor-5cc5bf6845-57s2f\" (UID: \"ab659d35-5ddd-4283-8bc8-e3b39f98d61c\") " pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"
Jul 28 18:03:04.371152 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.371129    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-kube-rbac-proxy-sar-config\") pod \"isvc-primary-e02d20-predictor-5cc5bf6845-57s2f\" (UID: \"ab659d35-5ddd-4283-8bc8-e3b39f98d61c\") " pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"
Jul 28 18:03:04.371194 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.371165    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-proxy-tls\") pod \"isvc-primary-e02d20-predictor-5cc5bf6845-57s2f\" (UID: \"ab659d35-5ddd-4283-8bc8-e3b39f98d61c\") " pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"
Jul 28 18:03:04.371252 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.371209    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzw22\" (UniqueName: \"kubernetes.io/projected/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-kube-api-access-zzw22\") pod \"isvc-primary-e02d20-predictor-5cc5bf6845-57s2f\" (UID: \"ab659d35-5ddd-4283-8bc8-e3b39f98d61c\") " pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"
Jul 28 18:03:04.371385 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.371362    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-kserve-provision-location\") pod \"isvc-primary-e02d20-predictor-5cc5bf6845-57s2f\" (UID: \"ab659d35-5ddd-4283-8bc8-e3b39f98d61c\") " pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"
Jul 28 18:03:04.371714 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.371692    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-cabundle-cert\") pod \"isvc-primary-e02d20-predictor-5cc5bf6845-57s2f\" (UID: \"ab659d35-5ddd-4283-8bc8-e3b39f98d61c\") " pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"
Jul 28 18:03:04.371808 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.371786    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-kube-rbac-proxy-sar-config\") pod \"isvc-primary-e02d20-predictor-5cc5bf6845-57s2f\" (UID: \"ab659d35-5ddd-4283-8bc8-e3b39f98d61c\") " pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"
Jul 28 18:03:04.373837 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.373820    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-proxy-tls\") pod \"isvc-primary-e02d20-predictor-5cc5bf6845-57s2f\" (UID: \"ab659d35-5ddd-4283-8bc8-e3b39f98d61c\") " pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"
Jul 28 18:03:04.388126 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.388098    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzw22\" (UniqueName: \"kubernetes.io/projected/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-kube-api-access-zzw22\") pod \"isvc-primary-e02d20-predictor-5cc5bf6845-57s2f\" (UID: \"ab659d35-5ddd-4283-8bc8-e3b39f98d61c\") " pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"
Jul 28 18:03:04.492297 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.492252    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"
Jul 28 18:03:04.623690 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.623568    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"]
Jul 28 18:03:04.626304 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:03:04.626277    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab659d35_5ddd_4283_8bc8_e3b39f98d61c.slice/crio-22527801b4ccc89c5362e7b147691ca460045040f6705baa2855f7d9dbdec4c8 WatchSource:0}: Error finding container 22527801b4ccc89c5362e7b147691ca460045040f6705baa2855f7d9dbdec4c8: Status 404 returned error can't find the container with id 22527801b4ccc89c5362e7b147691ca460045040f6705baa2855f7d9dbdec4c8
Jul 28 18:03:04.920839 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.920799    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f" event={"ID":"ab659d35-5ddd-4283-8bc8-e3b39f98d61c","Type":"ContainerStarted","Data":"c79b453874b7dc80def92dbbf2825963777819719886720654529a3142f14460"}
Jul 28 18:03:04.921047 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:04.920885    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f" event={"ID":"ab659d35-5ddd-4283-8bc8-e3b39f98d61c","Type":"ContainerStarted","Data":"22527801b4ccc89c5362e7b147691ca460045040f6705baa2855f7d9dbdec4c8"}
Jul 28 18:03:08.939642 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:08.939593    2581 generic.go:356] "Generic (PLEG): container finished" podID="ab659d35-5ddd-4283-8bc8-e3b39f98d61c" containerID="c79b453874b7dc80def92dbbf2825963777819719886720654529a3142f14460" exitCode=0
Jul 28 18:03:08.940095 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:08.939671    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f" event={"ID":"ab659d35-5ddd-4283-8bc8-e3b39f98d61c","Type":"ContainerDied","Data":"c79b453874b7dc80def92dbbf2825963777819719886720654529a3142f14460"}
Jul 28 18:03:09.945357 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:09.945318    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f" event={"ID":"ab659d35-5ddd-4283-8bc8-e3b39f98d61c","Type":"ContainerStarted","Data":"fe001861c5d670f9273ba280e4de5c62aca0bae3d471d9b8726393d4572c7fd2"}
Jul 28 18:03:09.945768 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:09.945382    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f" event={"ID":"ab659d35-5ddd-4283-8bc8-e3b39f98d61c","Type":"ContainerStarted","Data":"85bb2b2cadee3421f318c4ab59a1d893cdaeb39e4157d842499acb9529c99be2"}
Jul 28 18:03:09.945768 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:09.945666    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"
Jul 28 18:03:09.945849 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:09.945795    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"
Jul 28 18:03:09.947223 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:09.947198    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f" podUID="ab659d35-5ddd-4283-8bc8-e3b39f98d61c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 18:03:09.969935 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:09.969880    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f" podStartSLOduration=2.169139121 podStartE2EDuration="5.969862746s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.800723625s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:03:09.967804979 +0000 UTC m=+3238.242421414" watchObservedRunningTime="2026-07-28 18:03:09.969862746 +0000 UTC m=+3238.244479178"
Jul 28 18:03:10.949079 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:10.949040    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f" podUID="ab659d35-5ddd-4283-8bc8-e3b39f98d61c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 18:03:15.953561 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:15.953528    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"
Jul 28 18:03:15.954094 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:15.954065    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f" podUID="ab659d35-5ddd-4283-8bc8-e3b39f98d61c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 18:03:25.954478 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:25.954439    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f" podUID="ab659d35-5ddd-4283-8bc8-e3b39f98d61c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 18:03:35.954182 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:35.954140    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f" podUID="ab659d35-5ddd-4283-8bc8-e3b39f98d61c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 18:03:45.954223 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:45.954181    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f" podUID="ab659d35-5ddd-4283-8bc8-e3b39f98d61c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 18:03:55.954814 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:03:55.954773    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f" podUID="ab659d35-5ddd-4283-8bc8-e3b39f98d61c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 18:04:05.954539 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:05.954497    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f" podUID="ab659d35-5ddd-4283-8bc8-e3b39f98d61c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 18:04:12.379529 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:12.379491    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 18:04:12.381868 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:12.381845    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 18:04:15.954187 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:15.954142    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f" podUID="ab659d35-5ddd-4283-8bc8-e3b39f98d61c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 18:04:25.954787 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:25.954750    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"
Jul 28 18:04:26.953664 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:26.953629    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:49.399802 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:49.399762    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:49.513691 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:49.513653    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc"]
Jul 28 18:04:49.516907 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:49.516890    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc"
Jul 28 18:04:49.520121 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:49.519969    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-e02d20-kube-rbac-proxy-sar-config\""
Jul 28 18:04:49.520121 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:49.519988    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-e02d20\""
Jul 28 18:04:49.520121 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:49.520086    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-e02d20-dockercfg-h4nqx\""
Jul 28 18:04:49.520349 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:49.520234    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-e02d20-predictor-serving-cert\""
Jul 28 18:04:49.533890 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:49.533855    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc"]
Jul 28 18:04:49.623202 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:49.623154    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:04:49.653899 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:49.653813    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/953c7648-14f1-4793-b960-f22173b07a0f-proxy-tls\") pod \"isvc-secondary-e02d20-predictor-568d6748f4-gpmtc\" (UID: \"953c7648-14f1-4793-b960-f22173b07a0f\") " pod="kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc"
Jul 28 18:04:49.653899 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:49.653859    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/953c7648-14f1-4793-b960-f22173b07a0f-cabundle-cert\") pod \"isvc-secondary-e02d20-predictor-568d6748f4-gpmtc\" (UID: \"953c7648-14f1-4793-b960-f22173b07a0f\") " pod="kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc"
Jul 28 18:04:49.653899 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:49.653883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/953c7648-14f1-4793-b960-f22173b07a0f-kserve-provision-location\") pod \"isvc-secondary-e02d20-predictor-568d6748f4-gpmtc\" (UID: \"953c7648-14f1-4793-b960-f22173b07a0f\") " pod="kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc"
Jul 28 18:04:49.654140 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:49.653933    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/953c7648-14f1-4793-b960-f22173b07a0f-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-e02d20-predictor-568d6748f4-gpmtc\" (UID: \"953c7648-14f1-4793-b960-f22173b07a0f\") " pod="kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc"
Jul 28 18:04:49.654140 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:49.653987    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llzc6\" (UniqueName: \"kubernetes.io/projected/953c7648-14f1-4793-b960-f22173b07a0f-kube-api-access-llzc6\") pod \"isvc-secondary-e02d20-predictor-568d6748f4-gpmtc\" (UID: \"953c7648-14f1-4793-b960-f22173b07a0f\") " pod="kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc"
Jul 28 18:04:49.755216 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:49.755176    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/953c7648-14f1-4793-b960-f22173b07a0f-proxy-tls\") pod \"isvc-secondary-e02d20-predictor-568d6748f4-gpmtc\" (UID: \"953c7648-14f1-4793-b960-f22173b07a0f\") " pod="kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc"
Jul 28 18:04:49.755399 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:49.755225    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/953c7648-14f1-4793-b960-f22173b07a0f-cabundle-cert\") pod \"isvc-secondary-e02d20-predictor-568d6748f4-gpmtc\" (UID: \"953c7648-14f1-4793-b960-f22173b07a0f\") " pod="kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc"
Jul 28 18:04:49.755399 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:49.755339    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/953c7648-14f1-4793-b960-f22173b07a0f-kserve-provision-location\") pod \"isvc-secondary-e02d20-predictor-568d6748f4-gpmtc\" (UID: \"953c7648-14f1-4793-b960-f22173b07a0f\") " pod="kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc"
Jul 28 18:04:49.755399 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:49.755396    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/953c7648-14f1-4793-b960-f22173b07a0f-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-e02d20-predictor-568d6748f4-gpmtc\" (UID: \"953c7648-14f1-4793-b960-f22173b07a0f\") " pod="kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc"
Jul 28 18:04:49.755570 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:49.755433    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llzc6\" (UniqueName: \"kubernetes.io/projected/953c7648-14f1-4793-b960-f22173b07a0f-kube-api-access-llzc6\") pod \"isvc-secondary-e02d20-predictor-568d6748f4-gpmtc\" (UID: \"953c7648-14f1-4793-b960-f22173b07a0f\") " pod="kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc"
Jul 28 18:04:49.755840 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:49.755809    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/953c7648-14f1-4793-b960-f22173b07a0f-kserve-provision-location\") pod \"isvc-secondary-e02d20-predictor-568d6748f4-gpmtc\" (UID: \"953c7648-14f1-4793-b960-f22173b07a0f\") " pod="kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc"
Jul 28 18:04:49.755988 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:49.755932    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/953c7648-14f1-4793-b960-f22173b07a0f-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-e02d20-predictor-568d6748f4-gpmtc\" (UID: \"953c7648-14f1-4793-b960-f22173b07a0f\") " pod="kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc"
Jul 28 18:04:49.756056 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:49.755964    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/953c7648-14f1-4793-b960-f22173b07a0f-cabundle-cert\") pod \"isvc-secondary-e02d20-predictor-568d6748f4-gpmtc\" (UID: \"953c7648-14f1-4793-b960-f22173b07a0f\") " pod="kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc"
Jul 28 18:04:49.757892 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:49.757867    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/953c7648-14f1-4793-b960-f22173b07a0f-proxy-tls\") pod \"isvc-secondary-e02d20-predictor-568d6748f4-gpmtc\" (UID: \"953c7648-14f1-4793-b960-f22173b07a0f\") " pod="kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc"
Jul 28 18:04:49.765270 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:49.765247    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llzc6\" (UniqueName: \"kubernetes.io/projected/953c7648-14f1-4793-b960-f22173b07a0f-kube-api-access-llzc6\") pod \"isvc-secondary-e02d20-predictor-568d6748f4-gpmtc\" (UID: \"953c7648-14f1-4793-b960-f22173b07a0f\") " pod="kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc"
Jul 28 18:04:49.827139 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:49.827100    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc"
Jul 28 18:04:49.962592 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:49.962564    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc"]
Jul 28 18:04:49.965186 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:04:49.965158    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod953c7648_14f1_4793_b960_f22173b07a0f.slice/crio-bfe1559acd554e7a03f73d6cea1d970c79bb2f72e67d8a8b605c1b2ac233e340 WatchSource:0}: Error finding container bfe1559acd554e7a03f73d6cea1d970c79bb2f72e67d8a8b605c1b2ac233e340: Status 404 returned error can't find the container with id bfe1559acd554e7a03f73d6cea1d970c79bb2f72e67d8a8b605c1b2ac233e340
Jul 28 18:04:50.291167 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:50.291079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc" event={"ID":"953c7648-14f1-4793-b960-f22173b07a0f","Type":"ContainerStarted","Data":"fabd0766f35d7370fecc9d2c0adc6f6df9b456eb52f8a5ec4f12766706010c11"}
Jul 28 18:04:50.291167 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:50.291143    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc" event={"ID":"953c7648-14f1-4793-b960-f22173b07a0f","Type":"ContainerStarted","Data":"bfe1559acd554e7a03f73d6cea1d970c79bb2f72e67d8a8b605c1b2ac233e340"}
Jul 28 18:04:54.306873 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:54.306840    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-e02d20-predictor-568d6748f4-gpmtc_953c7648-14f1-4793-b960-f22173b07a0f/storage-initializer/0.log"
Jul 28 18:04:54.307303 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:54.306882    2581 generic.go:356] "Generic (PLEG): container finished" podID="953c7648-14f1-4793-b960-f22173b07a0f" containerID="fabd0766f35d7370fecc9d2c0adc6f6df9b456eb52f8a5ec4f12766706010c11" exitCode=1
Jul 28 18:04:54.307303 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:54.306911    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc" event={"ID":"953c7648-14f1-4793-b960-f22173b07a0f","Type":"ContainerDied","Data":"fabd0766f35d7370fecc9d2c0adc6f6df9b456eb52f8a5ec4f12766706010c11"}
Jul 28 18:04:55.312293 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:55.312263    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-e02d20-predictor-568d6748f4-gpmtc_953c7648-14f1-4793-b960-f22173b07a0f/storage-initializer/0.log"
Jul 28 18:04:55.312831 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:55.312366    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc" event={"ID":"953c7648-14f1-4793-b960-f22173b07a0f","Type":"ContainerStarted","Data":"7fa2db7aa8edfb3fd5b2f930f9b63a860b25c8be389dd197b2785e0ccce14433"}
Jul 28 18:04:59.326275 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:59.326194    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-e02d20-predictor-568d6748f4-gpmtc_953c7648-14f1-4793-b960-f22173b07a0f/storage-initializer/1.log"
Jul 28 18:04:59.326667 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:59.326581    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-e02d20-predictor-568d6748f4-gpmtc_953c7648-14f1-4793-b960-f22173b07a0f/storage-initializer/0.log"
Jul 28 18:04:59.326667 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:59.326610    2581 generic.go:356] "Generic (PLEG): container finished" podID="953c7648-14f1-4793-b960-f22173b07a0f" containerID="7fa2db7aa8edfb3fd5b2f930f9b63a860b25c8be389dd197b2785e0ccce14433" exitCode=1
Jul 28 18:04:59.326759 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:59.326676    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc" event={"ID":"953c7648-14f1-4793-b960-f22173b07a0f","Type":"ContainerDied","Data":"7fa2db7aa8edfb3fd5b2f930f9b63a860b25c8be389dd197b2785e0ccce14433"}
Jul 28 18:04:59.326759 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:59.326733    2581 scope.go:117] "RemoveContainer" containerID="fabd0766f35d7370fecc9d2c0adc6f6df9b456eb52f8a5ec4f12766706010c11"
Jul 28 18:04:59.327121 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:59.327097    2581 scope.go:117] "RemoveContainer" containerID="fabd0766f35d7370fecc9d2c0adc6f6df9b456eb52f8a5ec4f12766706010c11"
Jul 28 18:04:59.336984 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:04:59.336930    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-e02d20-predictor-568d6748f4-gpmtc_kserve-ci-e2e-test(953c7648-14f1-4793-b960-f22173b07a0f)\"" pod="kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc" podUID="953c7648-14f1-4793-b960-f22173b07a0f"
Jul 28 18:04:59.337189 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:04:59.337166    2581 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-e02d20-predictor-568d6748f4-gpmtc_kserve-ci-e2e-test_953c7648-14f1-4793-b960-f22173b07a0f_0 in pod sandbox bfe1559acd554e7a03f73d6cea1d970c79bb2f72e67d8a8b605c1b2ac233e340 from index: no such id: 'fabd0766f35d7370fecc9d2c0adc6f6df9b456eb52f8a5ec4f12766706010c11'" containerID="fabd0766f35d7370fecc9d2c0adc6f6df9b456eb52f8a5ec4f12766706010c11"
Jul 28 18:04:59.337253 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:04:59.337201    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fabd0766f35d7370fecc9d2c0adc6f6df9b456eb52f8a5ec4f12766706010c11"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-e02d20-predictor-568d6748f4-gpmtc_kserve-ci-e2e-test_953c7648-14f1-4793-b960-f22173b07a0f_0 in pod sandbox bfe1559acd554e7a03f73d6cea1d970c79bb2f72e67d8a8b605c1b2ac233e340 from index: no such id: 'fabd0766f35d7370fecc9d2c0adc6f6df9b456eb52f8a5ec4f12766706010c11'"
Jul 28 18:05:00.331329 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:00.331298    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-e02d20-predictor-568d6748f4-gpmtc_953c7648-14f1-4793-b960-f22173b07a0f/storage-initializer/1.log"
Jul 28 18:05:05.539586 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:05.539552    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc"]
Jul 28 18:05:05.560570 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:05.560535    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:05:05.632309 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:05.632276    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"]
Jul 28 18:05:05.632569 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:05.632540    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f" podUID="ab659d35-5ddd-4283-8bc8-e3b39f98d61c" containerName="kserve-container" containerID="cri-o://85bb2b2cadee3421f318c4ab59a1d893cdaeb39e4157d842499acb9529c99be2" gracePeriod=30
Jul 28 18:05:05.632675 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:05.632634    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f" podUID="ab659d35-5ddd-4283-8bc8-e3b39f98d61c" containerName="kube-rbac-proxy" containerID="cri-o://fe001861c5d670f9273ba280e4de5c62aca0bae3d471d9b8726393d4572c7fd2" gracePeriod=30
Jul 28 18:05:05.659086 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:05.659044    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:05:05.751078 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:05.751056    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-e02d20-predictor-568d6748f4-gpmtc_953c7648-14f1-4793-b960-f22173b07a0f/storage-initializer/1.log"
Jul 28 18:05:05.753261 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:05.753233    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc"
Jul 28 18:05:05.774721 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:05.774685    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/953c7648-14f1-4793-b960-f22173b07a0f-kserve-provision-location\") pod \"953c7648-14f1-4793-b960-f22173b07a0f\" (UID: \"953c7648-14f1-4793-b960-f22173b07a0f\") "
Jul 28 18:05:05.774721 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:05.774724    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/953c7648-14f1-4793-b960-f22173b07a0f-cabundle-cert\") pod \"953c7648-14f1-4793-b960-f22173b07a0f\" (UID: \"953c7648-14f1-4793-b960-f22173b07a0f\") "
Jul 28 18:05:05.775007 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:05.774757    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llzc6\" (UniqueName: \"kubernetes.io/projected/953c7648-14f1-4793-b960-f22173b07a0f-kube-api-access-llzc6\") pod \"953c7648-14f1-4793-b960-f22173b07a0f\" (UID: \"953c7648-14f1-4793-b960-f22173b07a0f\") "
Jul 28 18:05:05.775007 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:05.774796    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/953c7648-14f1-4793-b960-f22173b07a0f-proxy-tls\") pod \"953c7648-14f1-4793-b960-f22173b07a0f\" (UID: \"953c7648-14f1-4793-b960-f22173b07a0f\") "
Jul 28 18:05:05.775007 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:05.774817    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/953c7648-14f1-4793-b960-f22173b07a0f-kube-rbac-proxy-sar-config\") pod \"953c7648-14f1-4793-b960-f22173b07a0f\" (UID: \"953c7648-14f1-4793-b960-f22173b07a0f\") "
Jul 28 18:05:05.775176 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:05.775075    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/953c7648-14f1-4793-b960-f22173b07a0f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "953c7648-14f1-4793-b960-f22173b07a0f" (UID: "953c7648-14f1-4793-b960-f22173b07a0f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:05:05.775239 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:05.775182    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/953c7648-14f1-4793-b960-f22173b07a0f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "953c7648-14f1-4793-b960-f22173b07a0f" (UID: "953c7648-14f1-4793-b960-f22173b07a0f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:05:05.775239 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:05.775229    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/953c7648-14f1-4793-b960-f22173b07a0f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "953c7648-14f1-4793-b960-f22173b07a0f" (UID: "953c7648-14f1-4793-b960-f22173b07a0f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:05:05.777055 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:05.777032    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/953c7648-14f1-4793-b960-f22173b07a0f-kube-api-access-llzc6" (OuterVolumeSpecName: "kube-api-access-llzc6") pod "953c7648-14f1-4793-b960-f22173b07a0f" (UID: "953c7648-14f1-4793-b960-f22173b07a0f"). InnerVolumeSpecName "kube-api-access-llzc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:05:05.777140 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:05.777068    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/953c7648-14f1-4793-b960-f22173b07a0f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "953c7648-14f1-4793-b960-f22173b07a0f" (UID: "953c7648-14f1-4793-b960-f22173b07a0f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:05:05.875521 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:05.875421    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/953c7648-14f1-4793-b960-f22173b07a0f-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:05:05.875521 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:05.875457    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/953c7648-14f1-4793-b960-f22173b07a0f-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:05:05.875521 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:05.875468    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llzc6\" (UniqueName: \"kubernetes.io/projected/953c7648-14f1-4793-b960-f22173b07a0f-kube-api-access-llzc6\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:05:05.875521 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:05.875477    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/953c7648-14f1-4793-b960-f22173b07a0f-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:05:05.875521 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:05.875487    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/953c7648-14f1-4793-b960-f22173b07a0f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:05:05.949877 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:05.949825    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f" podUID="ab659d35-5ddd-4283-8bc8-e3b39f98d61c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Jul 28 18:05:05.954199 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:05.954165    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f" podUID="ab659d35-5ddd-4283-8bc8-e3b39f98d61c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 18:05:06.356613 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:06.356574    2581 generic.go:356] "Generic (PLEG): container finished" podID="ab659d35-5ddd-4283-8bc8-e3b39f98d61c" containerID="fe001861c5d670f9273ba280e4de5c62aca0bae3d471d9b8726393d4572c7fd2" exitCode=2
Jul 28 18:05:06.356791 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:06.356654    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f" event={"ID":"ab659d35-5ddd-4283-8bc8-e3b39f98d61c","Type":"ContainerDied","Data":"fe001861c5d670f9273ba280e4de5c62aca0bae3d471d9b8726393d4572c7fd2"}
Jul 28 18:05:06.357817 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:06.357800    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-e02d20-predictor-568d6748f4-gpmtc_953c7648-14f1-4793-b960-f22173b07a0f/storage-initializer/1.log"
Jul 28 18:05:06.357927 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:06.357893    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc"
Jul 28 18:05:06.357927 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:06.357893    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc" event={"ID":"953c7648-14f1-4793-b960-f22173b07a0f","Type":"ContainerDied","Data":"bfe1559acd554e7a03f73d6cea1d970c79bb2f72e67d8a8b605c1b2ac233e340"}
Jul 28 18:05:06.358005 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:06.357968    2581 scope.go:117] "RemoveContainer" containerID="7fa2db7aa8edfb3fd5b2f930f9b63a860b25c8be389dd197b2785e0ccce14433"
Jul 28 18:05:06.399873 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:06.399840    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc"]
Jul 28 18:05:06.405823 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:06.405795    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-e02d20-predictor-568d6748f4-gpmtc"]
Jul 28 18:05:08.289750 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:08.289716    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="953c7648-14f1-4793-b960-f22173b07a0f" path="/var/lib/kubelet/pods/953c7648-14f1-4793-b960-f22173b07a0f/volumes"
Jul 28 18:05:10.375384 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.375347    2581 generic.go:356] "Generic (PLEG): container finished" podID="ab659d35-5ddd-4283-8bc8-e3b39f98d61c" containerID="85bb2b2cadee3421f318c4ab59a1d893cdaeb39e4157d842499acb9529c99be2" exitCode=0
Jul 28 18:05:10.375720 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.375383    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f" event={"ID":"ab659d35-5ddd-4283-8bc8-e3b39f98d61c","Type":"ContainerDied","Data":"85bb2b2cadee3421f318c4ab59a1d893cdaeb39e4157d842499acb9529c99be2"}
Jul 28 18:05:10.375720 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.375445    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f" event={"ID":"ab659d35-5ddd-4283-8bc8-e3b39f98d61c","Type":"ContainerDied","Data":"22527801b4ccc89c5362e7b147691ca460045040f6705baa2855f7d9dbdec4c8"}
Jul 28 18:05:10.375720 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.375466    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22527801b4ccc89c5362e7b147691ca460045040f6705baa2855f7d9dbdec4c8"
Jul 28 18:05:10.384260 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.384241    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"
Jul 28 18:05:10.411204 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.411129    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-kserve-provision-location\") pod \"ab659d35-5ddd-4283-8bc8-e3b39f98d61c\" (UID: \"ab659d35-5ddd-4283-8bc8-e3b39f98d61c\") "
Jul 28 18:05:10.411204 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.411164    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzw22\" (UniqueName: \"kubernetes.io/projected/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-kube-api-access-zzw22\") pod \"ab659d35-5ddd-4283-8bc8-e3b39f98d61c\" (UID: \"ab659d35-5ddd-4283-8bc8-e3b39f98d61c\") "
Jul 28 18:05:10.411204 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.411187    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-proxy-tls\") pod \"ab659d35-5ddd-4283-8bc8-e3b39f98d61c\" (UID: \"ab659d35-5ddd-4283-8bc8-e3b39f98d61c\") "
Jul 28 18:05:10.411459 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.411214    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-cabundle-cert\") pod \"ab659d35-5ddd-4283-8bc8-e3b39f98d61c\" (UID: \"ab659d35-5ddd-4283-8bc8-e3b39f98d61c\") "
Jul 28 18:05:10.411459 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.411247    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-kube-rbac-proxy-sar-config\") pod \"ab659d35-5ddd-4283-8bc8-e3b39f98d61c\" (UID: \"ab659d35-5ddd-4283-8bc8-e3b39f98d61c\") "
Jul 28 18:05:10.411564 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.411481    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ab659d35-5ddd-4283-8bc8-e3b39f98d61c" (UID: "ab659d35-5ddd-4283-8bc8-e3b39f98d61c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:05:10.411661 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.411637    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ab659d35-5ddd-4283-8bc8-e3b39f98d61c" (UID: "ab659d35-5ddd-4283-8bc8-e3b39f98d61c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:05:10.411705 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.411642    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ab659d35-5ddd-4283-8bc8-e3b39f98d61c" (UID: "ab659d35-5ddd-4283-8bc8-e3b39f98d61c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:05:10.413267 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.413244    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-kube-api-access-zzw22" (OuterVolumeSpecName: "kube-api-access-zzw22") pod "ab659d35-5ddd-4283-8bc8-e3b39f98d61c" (UID: "ab659d35-5ddd-4283-8bc8-e3b39f98d61c"). InnerVolumeSpecName "kube-api-access-zzw22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:05:10.413320 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.413271    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ab659d35-5ddd-4283-8bc8-e3b39f98d61c" (UID: "ab659d35-5ddd-4283-8bc8-e3b39f98d61c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:05:10.512194 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.512160    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:05:10.512194 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.512191    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzw22\" (UniqueName: \"kubernetes.io/projected/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-kube-api-access-zzw22\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:05:10.512194 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.512204    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:05:10.512452 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.512217    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:05:10.512452 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.512229    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab659d35-5ddd-4283-8bc8-e3b39f98d61c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:05:10.714144 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.714050    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:05:10.781718 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.781678    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn"]
Jul 28 18:05:10.781980 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.781965    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab659d35-5ddd-4283-8bc8-e3b39f98d61c" containerName="kube-rbac-proxy"
Jul 28 18:05:10.781980 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.781980    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab659d35-5ddd-4283-8bc8-e3b39f98d61c" containerName="kube-rbac-proxy"
Jul 28 18:05:10.782110 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.781994    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab659d35-5ddd-4283-8bc8-e3b39f98d61c" containerName="storage-initializer"
Jul 28 18:05:10.782110 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.782000    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab659d35-5ddd-4283-8bc8-e3b39f98d61c" containerName="storage-initializer"
Jul 28 18:05:10.782110 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.782009    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab659d35-5ddd-4283-8bc8-e3b39f98d61c" containerName="kserve-container"
Jul 28 18:05:10.782110 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.782015    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab659d35-5ddd-4283-8bc8-e3b39f98d61c" containerName="kserve-container"
Jul 28 18:05:10.782110 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.782021    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="953c7648-14f1-4793-b960-f22173b07a0f" containerName="storage-initializer"
Jul 28 18:05:10.782110 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.782027    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="953c7648-14f1-4793-b960-f22173b07a0f" containerName="storage-initializer"
Jul 28 18:05:10.782110 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.782035    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="953c7648-14f1-4793-b960-f22173b07a0f" containerName="storage-initializer"
Jul 28 18:05:10.782110 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.782040    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="953c7648-14f1-4793-b960-f22173b07a0f" containerName="storage-initializer"
Jul 28 18:05:10.782110 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.782078    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="953c7648-14f1-4793-b960-f22173b07a0f" containerName="storage-initializer"
Jul 28 18:05:10.782110 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.782085    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab659d35-5ddd-4283-8bc8-e3b39f98d61c" containerName="kserve-container"
Jul 28 18:05:10.782110 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.782092    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab659d35-5ddd-4283-8bc8-e3b39f98d61c" containerName="kube-rbac-proxy"
Jul 28 18:05:10.782656 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.782166    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="953c7648-14f1-4793-b960-f22173b07a0f" containerName="storage-initializer"
Jul 28 18:05:10.785167 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.785148    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn"
Jul 28 18:05:10.788375 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.788350    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-673891-kube-rbac-proxy-sar-config\""
Jul 28 18:05:10.788540 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.788350    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-673891-dockercfg-bsq7h\""
Jul 28 18:05:10.788540 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.788350    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-673891-predictor-serving-cert\""
Jul 28 18:05:10.788917 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.788897    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-673891\""
Jul 28 18:05:10.802056 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.802024    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn"]
Jul 28 18:05:10.814062 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.814033    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxcqw\" (UniqueName: \"kubernetes.io/projected/913be5bb-f53e-4eff-8db0-80748adbde97-kube-api-access-wxcqw\") pod \"isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn\" (UID: \"913be5bb-f53e-4eff-8db0-80748adbde97\") " pod="kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn"
Jul 28 18:05:10.814205 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.814086    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/913be5bb-f53e-4eff-8db0-80748adbde97-cabundle-cert\") pod \"isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn\" (UID: \"913be5bb-f53e-4eff-8db0-80748adbde97\") " pod="kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn"
Jul 28 18:05:10.814205 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.814162    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/913be5bb-f53e-4eff-8db0-80748adbde97-kserve-provision-location\") pod \"isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn\" (UID: \"913be5bb-f53e-4eff-8db0-80748adbde97\") " pod="kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn"
Jul 28 18:05:10.814322 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.814244    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/913be5bb-f53e-4eff-8db0-80748adbde97-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn\" (UID: \"913be5bb-f53e-4eff-8db0-80748adbde97\") " pod="kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn"
Jul 28 18:05:10.814322 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.814280    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/913be5bb-f53e-4eff-8db0-80748adbde97-proxy-tls\") pod \"isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn\" (UID: \"913be5bb-f53e-4eff-8db0-80748adbde97\") " pod="kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn"
Jul 28 18:05:10.914758 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.914714    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxcqw\" (UniqueName: \"kubernetes.io/projected/913be5bb-f53e-4eff-8db0-80748adbde97-kube-api-access-wxcqw\") pod \"isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn\" (UID: \"913be5bb-f53e-4eff-8db0-80748adbde97\") " pod="kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn"
Jul 28 18:05:10.914977 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.914774    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/913be5bb-f53e-4eff-8db0-80748adbde97-cabundle-cert\") pod \"isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn\" (UID: \"913be5bb-f53e-4eff-8db0-80748adbde97\") " pod="kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn"
Jul 28 18:05:10.914977 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.914825    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/913be5bb-f53e-4eff-8db0-80748adbde97-kserve-provision-location\") pod \"isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn\" (UID: \"913be5bb-f53e-4eff-8db0-80748adbde97\") " pod="kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn"
Jul 28 18:05:10.914977 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.914882    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/913be5bb-f53e-4eff-8db0-80748adbde97-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn\" (UID: \"913be5bb-f53e-4eff-8db0-80748adbde97\") " pod="kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn"
Jul 28 18:05:10.914977 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.914908    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/913be5bb-f53e-4eff-8db0-80748adbde97-proxy-tls\") pod \"isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn\" (UID: \"913be5bb-f53e-4eff-8db0-80748adbde97\") " pod="kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn"
Jul 28 18:05:10.915212 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:05:10.915039    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-673891-predictor-serving-cert: secret "isvc-init-fail-673891-predictor-serving-cert" not found
Jul 28 18:05:10.915212 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:05:10.915111    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/913be5bb-f53e-4eff-8db0-80748adbde97-proxy-tls podName:913be5bb-f53e-4eff-8db0-80748adbde97 nodeName:}" failed. No retries permitted until 2026-07-28 18:05:11.415092951 +0000 UTC m=+3359.689709377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/913be5bb-f53e-4eff-8db0-80748adbde97-proxy-tls") pod "isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn" (UID: "913be5bb-f53e-4eff-8db0-80748adbde97") : secret "isvc-init-fail-673891-predictor-serving-cert" not found
Jul 28 18:05:10.915429 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.915403    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/913be5bb-f53e-4eff-8db0-80748adbde97-kserve-provision-location\") pod \"isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn\" (UID: \"913be5bb-f53e-4eff-8db0-80748adbde97\") " pod="kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn"
Jul 28 18:05:10.915580 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.915556    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/913be5bb-f53e-4eff-8db0-80748adbde97-cabundle-cert\") pod \"isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn\" (UID: \"913be5bb-f53e-4eff-8db0-80748adbde97\") " pod="kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn"
Jul 28 18:05:10.915644 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.915601    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/913be5bb-f53e-4eff-8db0-80748adbde97-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn\" (UID: \"913be5bb-f53e-4eff-8db0-80748adbde97\") " pod="kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn"
Jul 28 18:05:10.924383 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:10.924359    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxcqw\" (UniqueName: \"kubernetes.io/projected/913be5bb-f53e-4eff-8db0-80748adbde97-kube-api-access-wxcqw\") pod \"isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn\" (UID: \"913be5bb-f53e-4eff-8db0-80748adbde97\") " pod="kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn"
Jul 28 18:05:11.194994 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:11.194928    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:05:11.378980 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:11.378930    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"
Jul 28 18:05:11.404489 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:11.404454    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"]
Jul 28 18:05:11.409180 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:11.409152    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-e02d20-predictor-5cc5bf6845-57s2f"]
Jul 28 18:05:11.418646 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:11.418610    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/913be5bb-f53e-4eff-8db0-80748adbde97-proxy-tls\") pod \"isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn\" (UID: \"913be5bb-f53e-4eff-8db0-80748adbde97\") " pod="kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn"
Jul 28 18:05:11.421283 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:11.421256    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/913be5bb-f53e-4eff-8db0-80748adbde97-proxy-tls\") pod \"isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn\" (UID: \"913be5bb-f53e-4eff-8db0-80748adbde97\") " pod="kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn"
Jul 28 18:05:11.695410 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:11.695370    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn"
Jul 28 18:05:11.834425 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:11.834396    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn"]
Jul 28 18:05:11.836063 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:05:11.836036    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod913be5bb_f53e_4eff_8db0_80748adbde97.slice/crio-9ab7bf891ae8827e0696fd840201bd9d90219076bd064eef99e6d42cd8de4f4f WatchSource:0}: Error finding container 9ab7bf891ae8827e0696fd840201bd9d90219076bd064eef99e6d42cd8de4f4f: Status 404 returned error can't find the container with id 9ab7bf891ae8827e0696fd840201bd9d90219076bd064eef99e6d42cd8de4f4f
Jul 28 18:05:12.289430 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:12.289389    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab659d35-5ddd-4283-8bc8-e3b39f98d61c" path="/var/lib/kubelet/pods/ab659d35-5ddd-4283-8bc8-e3b39f98d61c/volumes"
Jul 28 18:05:12.383864 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:12.383827    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn" event={"ID":"913be5bb-f53e-4eff-8db0-80748adbde97","Type":"ContainerStarted","Data":"b882fdb957689502f37400bd51b597062e4767a498d32f0568c62d79dde35d6b"}
Jul 28 18:05:12.384299 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:12.383888    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn" event={"ID":"913be5bb-f53e-4eff-8db0-80748adbde97","Type":"ContainerStarted","Data":"9ab7bf891ae8827e0696fd840201bd9d90219076bd064eef99e6d42cd8de4f4f"}
Jul 28 18:05:18.407454 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:18.407420    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn_913be5bb-f53e-4eff-8db0-80748adbde97/storage-initializer/0.log"
Jul 28 18:05:18.407921 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:18.407466    2581 generic.go:356] "Generic (PLEG): container finished" podID="913be5bb-f53e-4eff-8db0-80748adbde97" containerID="b882fdb957689502f37400bd51b597062e4767a498d32f0568c62d79dde35d6b" exitCode=1
Jul 28 18:05:18.407921 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:18.407522    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn" event={"ID":"913be5bb-f53e-4eff-8db0-80748adbde97","Type":"ContainerDied","Data":"b882fdb957689502f37400bd51b597062e4767a498d32f0568c62d79dde35d6b"}
Jul 28 18:05:19.412718 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:19.412687    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn_913be5bb-f53e-4eff-8db0-80748adbde97/storage-initializer/0.log"
Jul 28 18:05:19.413217 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:19.412806    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn" event={"ID":"913be5bb-f53e-4eff-8db0-80748adbde97","Type":"ContainerStarted","Data":"89ae564e7911d535bd1193736b21196b20349de3dddcdf21d74bdd58195c363f"}
Jul 28 18:05:20.950744 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:20.950705    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn"]
Jul 28 18:05:20.951165 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:20.950909    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn" podUID="913be5bb-f53e-4eff-8db0-80748adbde97" containerName="storage-initializer" containerID="cri-o://89ae564e7911d535bd1193736b21196b20349de3dddcdf21d74bdd58195c363f" gracePeriod=30
Jul 28 18:05:20.985020 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:20.984986    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:05:24.888648 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:24.888625    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn_913be5bb-f53e-4eff-8db0-80748adbde97/storage-initializer/1.log"
Jul 28 18:05:24.889048 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:24.889033    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn_913be5bb-f53e-4eff-8db0-80748adbde97/storage-initializer/0.log"
Jul 28 18:05:24.891335 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:24.891321    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn"
Jul 28 18:05:24.923733 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:24.923645    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/913be5bb-f53e-4eff-8db0-80748adbde97-proxy-tls\") pod \"913be5bb-f53e-4eff-8db0-80748adbde97\" (UID: \"913be5bb-f53e-4eff-8db0-80748adbde97\") "
Jul 28 18:05:24.923733 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:24.923712    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxcqw\" (UniqueName: \"kubernetes.io/projected/913be5bb-f53e-4eff-8db0-80748adbde97-kube-api-access-wxcqw\") pod \"913be5bb-f53e-4eff-8db0-80748adbde97\" (UID: \"913be5bb-f53e-4eff-8db0-80748adbde97\") "
Jul 28 18:05:24.924041 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:24.923837    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/913be5bb-f53e-4eff-8db0-80748adbde97-kserve-provision-location\") pod \"913be5bb-f53e-4eff-8db0-80748adbde97\" (UID: \"913be5bb-f53e-4eff-8db0-80748adbde97\") "
Jul 28 18:05:24.924041 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:24.923898    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/913be5bb-f53e-4eff-8db0-80748adbde97-kube-rbac-proxy-sar-config\") pod \"913be5bb-f53e-4eff-8db0-80748adbde97\" (UID: \"913be5bb-f53e-4eff-8db0-80748adbde97\") "
Jul 28 18:05:24.924041 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:24.923972    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/913be5bb-f53e-4eff-8db0-80748adbde97-cabundle-cert\") pod \"913be5bb-f53e-4eff-8db0-80748adbde97\" (UID: \"913be5bb-f53e-4eff-8db0-80748adbde97\") "
Jul 28 18:05:24.924206 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:24.924183    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/913be5bb-f53e-4eff-8db0-80748adbde97-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "913be5bb-f53e-4eff-8db0-80748adbde97" (UID: "913be5bb-f53e-4eff-8db0-80748adbde97"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:05:24.924206 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:24.924238    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/913be5bb-f53e-4eff-8db0-80748adbde97-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "913be5bb-f53e-4eff-8db0-80748adbde97" (UID: "913be5bb-f53e-4eff-8db0-80748adbde97"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:05:24.924474 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:24.924454    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/913be5bb-f53e-4eff-8db0-80748adbde97-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "913be5bb-f53e-4eff-8db0-80748adbde97" (UID: "913be5bb-f53e-4eff-8db0-80748adbde97"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:05:24.925963 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:24.925926    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/913be5bb-f53e-4eff-8db0-80748adbde97-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "913be5bb-f53e-4eff-8db0-80748adbde97" (UID: "913be5bb-f53e-4eff-8db0-80748adbde97"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:05:24.925963 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:24.925932    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/913be5bb-f53e-4eff-8db0-80748adbde97-kube-api-access-wxcqw" (OuterVolumeSpecName: "kube-api-access-wxcqw") pod "913be5bb-f53e-4eff-8db0-80748adbde97" (UID: "913be5bb-f53e-4eff-8db0-80748adbde97"). InnerVolumeSpecName "kube-api-access-wxcqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:05:25.024842 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:25.024799    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/913be5bb-f53e-4eff-8db0-80748adbde97-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:05:25.024842 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:25.024836    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/913be5bb-f53e-4eff-8db0-80748adbde97-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:05:25.024842 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:25.024846    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/913be5bb-f53e-4eff-8db0-80748adbde97-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:05:25.025123 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:25.024854    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxcqw\" (UniqueName: \"kubernetes.io/projected/913be5bb-f53e-4eff-8db0-80748adbde97-kube-api-access-wxcqw\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:05:25.025123 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:25.024864    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/913be5bb-f53e-4eff-8db0-80748adbde97-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:05:25.434450 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:25.434422    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn_913be5bb-f53e-4eff-8db0-80748adbde97/storage-initializer/1.log"
Jul 28 18:05:25.434798 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:25.434782    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn_913be5bb-f53e-4eff-8db0-80748adbde97/storage-initializer/0.log"
Jul 28 18:05:25.434855 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:25.434834    2581 generic.go:356] "Generic (PLEG): container finished" podID="913be5bb-f53e-4eff-8db0-80748adbde97" containerID="89ae564e7911d535bd1193736b21196b20349de3dddcdf21d74bdd58195c363f" exitCode=1
Jul 28 18:05:25.434924 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:25.434874    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn" event={"ID":"913be5bb-f53e-4eff-8db0-80748adbde97","Type":"ContainerDied","Data":"89ae564e7911d535bd1193736b21196b20349de3dddcdf21d74bdd58195c363f"}
Jul 28 18:05:25.434997 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:25.434925    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn"
Jul 28 18:05:25.434997 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:25.434964    2581 scope.go:117] "RemoveContainer" containerID="89ae564e7911d535bd1193736b21196b20349de3dddcdf21d74bdd58195c363f"
Jul 28 18:05:25.435115 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:25.434929    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn" event={"ID":"913be5bb-f53e-4eff-8db0-80748adbde97","Type":"ContainerDied","Data":"9ab7bf891ae8827e0696fd840201bd9d90219076bd064eef99e6d42cd8de4f4f"}
Jul 28 18:05:25.443498 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:25.443480    2581 scope.go:117] "RemoveContainer" containerID="b882fdb957689502f37400bd51b597062e4767a498d32f0568c62d79dde35d6b"
Jul 28 18:05:25.450621 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:25.450605    2581 scope.go:117] "RemoveContainer" containerID="89ae564e7911d535bd1193736b21196b20349de3dddcdf21d74bdd58195c363f"
Jul 28 18:05:25.450854 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:05:25.450836    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89ae564e7911d535bd1193736b21196b20349de3dddcdf21d74bdd58195c363f\": container with ID starting with 89ae564e7911d535bd1193736b21196b20349de3dddcdf21d74bdd58195c363f not found: ID does not exist" containerID="89ae564e7911d535bd1193736b21196b20349de3dddcdf21d74bdd58195c363f"
Jul 28 18:05:25.450937 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:25.450866    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89ae564e7911d535bd1193736b21196b20349de3dddcdf21d74bdd58195c363f"} err="failed to get container status \"89ae564e7911d535bd1193736b21196b20349de3dddcdf21d74bdd58195c363f\": rpc error: code = NotFound desc = could not find container \"89ae564e7911d535bd1193736b21196b20349de3dddcdf21d74bdd58195c363f\": container with ID starting with 89ae564e7911d535bd1193736b21196b20349de3dddcdf21d74bdd58195c363f not found: ID does not exist"
Jul 28 18:05:25.450937 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:25.450891    2581 scope.go:117] "RemoveContainer" containerID="b882fdb957689502f37400bd51b597062e4767a498d32f0568c62d79dde35d6b"
Jul 28 18:05:25.451157 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:05:25.451139    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b882fdb957689502f37400bd51b597062e4767a498d32f0568c62d79dde35d6b\": container with ID starting with b882fdb957689502f37400bd51b597062e4767a498d32f0568c62d79dde35d6b not found: ID does not exist" containerID="b882fdb957689502f37400bd51b597062e4767a498d32f0568c62d79dde35d6b"
Jul 28 18:05:25.451194 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:25.451163    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b882fdb957689502f37400bd51b597062e4767a498d32f0568c62d79dde35d6b"} err="failed to get container status \"b882fdb957689502f37400bd51b597062e4767a498d32f0568c62d79dde35d6b\": rpc error: code = NotFound desc = could not find container \"b882fdb957689502f37400bd51b597062e4767a498d32f0568c62d79dde35d6b\": container with ID starting with b882fdb957689502f37400bd51b597062e4767a498d32f0568c62d79dde35d6b not found: ID does not exist"
Jul 28 18:05:25.487086 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:25.487050    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn"]
Jul 28 18:05:25.498584 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:25.498554    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-673891-predictor-849b4dcd5b-gjbgn"]
Jul 28 18:05:26.127784 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.127749    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"]
Jul 28 18:05:26.128204 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.128000    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="913be5bb-f53e-4eff-8db0-80748adbde97" containerName="storage-initializer"
Jul 28 18:05:26.128204 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.128011    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="913be5bb-f53e-4eff-8db0-80748adbde97" containerName="storage-initializer"
Jul 28 18:05:26.128204 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.128028    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="913be5bb-f53e-4eff-8db0-80748adbde97" containerName="storage-initializer"
Jul 28 18:05:26.128204 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.128034    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="913be5bb-f53e-4eff-8db0-80748adbde97" containerName="storage-initializer"
Jul 28 18:05:26.128204 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.128075    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="913be5bb-f53e-4eff-8db0-80748adbde97" containerName="storage-initializer"
Jul 28 18:05:26.128204 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.128155    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="913be5bb-f53e-4eff-8db0-80748adbde97" containerName="storage-initializer"
Jul 28 18:05:26.132652 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.132630    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"
Jul 28 18:05:26.135565 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.135536    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:05:26.141778 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.141757    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:05:26.143064 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.143050    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:05:26.143133 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.143078    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:05:26.143133 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.143078    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:05:26.145116 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.145100    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Jul 28 18:05:26.146510 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.146495    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Jul 28 18:05:26.146685 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.146669    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 18:05:26.153399 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.153375    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"]
Jul 28 18:05:26.234232 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.234191    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p5p2h\" (UID: \"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"
Jul 28 18:05:26.234232 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.234227    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p5p2h\" (UID: \"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"
Jul 28 18:05:26.234450 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.234254    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p5p2h\" (UID: \"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"
Jul 28 18:05:26.234450 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.234288    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p5p2h\" (UID: \"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"
Jul 28 18:05:26.234450 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.234310    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmc46\" (UniqueName: \"kubernetes.io/projected/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-kube-api-access-xmc46\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p5p2h\" (UID: \"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"
Jul 28 18:05:26.238794 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.238771    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:05:26.289543 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.289504    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="913be5bb-f53e-4eff-8db0-80748adbde97" path="/var/lib/kubelet/pods/913be5bb-f53e-4eff-8db0-80748adbde97/volumes"
Jul 28 18:05:26.335173 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.335141    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p5p2h\" (UID: \"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"
Jul 28 18:05:26.335173 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.335177    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p5p2h\" (UID: \"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"
Jul 28 18:05:26.335508 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.335206    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p5p2h\" (UID: \"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"
Jul 28 18:05:26.335508 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.335232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p5p2h\" (UID: \"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"
Jul 28 18:05:26.335508 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.335253    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmc46\" (UniqueName: \"kubernetes.io/projected/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-kube-api-access-xmc46\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p5p2h\" (UID: \"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"
Jul 28 18:05:26.335684 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.335592    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p5p2h\" (UID: \"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"
Jul 28 18:05:26.335961 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.335920    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p5p2h\" (UID: \"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"
Jul 28 18:05:26.336054 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.335993    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p5p2h\" (UID: \"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"
Jul 28 18:05:26.337770 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.337752    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p5p2h\" (UID: \"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"
Jul 28 18:05:26.348395 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.348364    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmc46\" (UniqueName: \"kubernetes.io/projected/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-kube-api-access-xmc46\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p5p2h\" (UID: \"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"
Jul 28 18:05:26.442518 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.442482    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"
Jul 28 18:05:26.585344 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:26.585307    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"]
Jul 28 18:05:26.588928 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:05:26.588884    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70dc3d5_4fe9_4865_b3d5_363efa8a2d9a.slice/crio-2be030cbb3dfbec782b94106554e408e30a243bb95795ca26f132d356e8485b6 WatchSource:0}: Error finding container 2be030cbb3dfbec782b94106554e408e30a243bb95795ca26f132d356e8485b6: Status 404 returned error can't find the container with id 2be030cbb3dfbec782b94106554e408e30a243bb95795ca26f132d356e8485b6
Jul 28 18:05:27.443874 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:27.443831    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h" event={"ID":"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a","Type":"ContainerStarted","Data":"6fec9ed5110ba1117f548e72b84ed837af5df661f1b190bbc3dd5b0fc60243ef"}
Jul 28 18:05:27.444302 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:27.443907    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h" event={"ID":"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a","Type":"ContainerStarted","Data":"2be030cbb3dfbec782b94106554e408e30a243bb95795ca26f132d356e8485b6"}
Jul 28 18:05:30.456378 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:30.456340    2581 generic.go:356] "Generic (PLEG): container finished" podID="f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" containerID="6fec9ed5110ba1117f548e72b84ed837af5df661f1b190bbc3dd5b0fc60243ef" exitCode=0
Jul 28 18:05:30.456794 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:30.456407    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h" event={"ID":"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a","Type":"ContainerDied","Data":"6fec9ed5110ba1117f548e72b84ed837af5df661f1b190bbc3dd5b0fc60243ef"}
Jul 28 18:05:30.457780 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:30.457762    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:05:51.558386 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:51.558347    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h" event={"ID":"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a","Type":"ContainerStarted","Data":"09f107141adbec11ab4a0b1e185e5e4bf20cae36cacb86181ebdd593acbb2a44"}
Jul 28 18:05:51.558900 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:51.558411    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h" event={"ID":"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a","Type":"ContainerStarted","Data":"9ab84b0e1104c6eceddda384c1d2145b88278d52114c3c32b4498f54a34cb69b"}
Jul 28 18:05:51.558900 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:51.558591    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"
Jul 28 18:05:52.563183 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:52.563148    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"
Jul 28 18:05:52.564275 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:52.564248    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h" podUID="f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 18:05:53.567216 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:53.567167    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h" podUID="f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 18:05:58.571585 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:58.571553    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"
Jul 28 18:05:58.572160 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:58.572134    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h" podUID="f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 18:05:58.608099 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:05:58.608039    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h" podStartSLOduration=8.513350625 podStartE2EDuration="32.608024262s" totalImagesPullingTime="20.49679487s" totalInitContainerRuntime="3.597878767s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:05:51.590964196 +0000 UTC m=+3399.865580627" watchObservedRunningTime="2026-07-28 18:05:58.608024262 +0000 UTC m=+3406.882640694"
Jul 28 18:06:08.572635 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:06:08.572551    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h" podUID="f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 18:06:18.572738 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:06:18.572691    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h" podUID="f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 18:06:28.572339 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:06:28.572300    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h" podUID="f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 18:06:38.572427 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:06:38.572377    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h" podUID="f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 18:06:48.572633 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:06:48.572583    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h" podUID="f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 18:06:58.572184 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:06:58.572143    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h" podUID="f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 18:07:08.573114 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:08.573072    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"
Jul 28 18:07:16.192978 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:16.192927    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"]
Jul 28 18:07:16.193536 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:16.193265    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h" podUID="f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" containerName="kserve-container" containerID="cri-o://9ab84b0e1104c6eceddda384c1d2145b88278d52114c3c32b4498f54a34cb69b" gracePeriod=30
Jul 28 18:07:16.193536 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:16.193318    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h" podUID="f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" containerName="kube-rbac-proxy" containerID="cri-o://09f107141adbec11ab4a0b1e185e5e4bf20cae36cacb86181ebdd593acbb2a44" gracePeriod=30
Jul 28 18:07:16.253801 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:16.253765    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:07:16.873562 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:16.873529    2581 generic.go:356] "Generic (PLEG): container finished" podID="f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" containerID="09f107141adbec11ab4a0b1e185e5e4bf20cae36cacb86181ebdd593acbb2a44" exitCode=2
Jul 28 18:07:16.873751 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:16.873609    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h" event={"ID":"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a","Type":"ContainerDied","Data":"09f107141adbec11ab4a0b1e185e5e4bf20cae36cacb86181ebdd593acbb2a44"}
Jul 28 18:07:18.568410 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:18.568357    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h" podUID="f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Jul 28 18:07:18.572214 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:18.572184    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h" podUID="f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 28 18:07:21.139400 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.139371    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"
Jul 28 18:07:21.237463 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.237364    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-kserve-provision-location\") pod \"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a\" (UID: \"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a\") "
Jul 28 18:07:21.237463 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.237415    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmc46\" (UniqueName: \"kubernetes.io/projected/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-kube-api-access-xmc46\") pod \"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a\" (UID: \"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a\") "
Jul 28 18:07:21.237463 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.237440    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-cabundle-cert\") pod \"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a\" (UID: \"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a\") "
Jul 28 18:07:21.237463 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.237465    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-proxy-tls\") pod \"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a\" (UID: \"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a\") "
Jul 28 18:07:21.237845 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.237504    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-kube-rbac-proxy-sar-config\") pod \"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a\" (UID: \"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a\") "
Jul 28 18:07:21.237919 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.237836    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" (UID: "f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:07:21.237919 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.237837    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" (UID: "f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:07:21.237919 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.237881    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" (UID: "f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:07:21.239968 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.239901    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" (UID: "f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:07:21.240107 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.239973    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-kube-api-access-xmc46" (OuterVolumeSpecName: "kube-api-access-xmc46") pod "f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" (UID: "f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a"). InnerVolumeSpecName "kube-api-access-xmc46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:07:21.290857 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.290819    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:07:21.338592 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.338552    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:07:21.338592 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.338586    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:07:21.338592 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.338595    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:07:21.338592 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.338604    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:07:21.338878 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.338615    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmc46\" (UniqueName: \"kubernetes.io/projected/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a-kube-api-access-xmc46\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:07:21.350146 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.350114    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"]
Jul 28 18:07:21.350358 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.350345    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" containerName="storage-initializer"
Jul 28 18:07:21.350405 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.350358    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" containerName="storage-initializer"
Jul 28 18:07:21.350405 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.350378    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" containerName="kserve-container"
Jul 28 18:07:21.350405 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.350384    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" containerName="kserve-container"
Jul 28 18:07:21.350405 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.350390    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" containerName="kube-rbac-proxy"
Jul 28 18:07:21.350405 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.350397    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" containerName="kube-rbac-proxy"
Jul 28 18:07:21.350585 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.350440    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" containerName="kube-rbac-proxy"
Jul 28 18:07:21.350585 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.350451    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" containerName="kserve-container"
Jul 28 18:07:21.353632 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.353615    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"
Jul 28 18:07:21.356248 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.356222    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Jul 28 18:07:21.356434 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.356414    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Jul 28 18:07:21.365294 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.365264    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"]
Jul 28 18:07:21.540008 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.539914    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c593963e-f5d8-4e09-8292-f6474dd368ac-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-m7l5h\" (UID: \"c593963e-f5d8-4e09-8292-f6474dd368ac\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"
Jul 28 18:07:21.540008 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.539962    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c593963e-f5d8-4e09-8292-f6474dd368ac-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-m7l5h\" (UID: \"c593963e-f5d8-4e09-8292-f6474dd368ac\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"
Jul 28 18:07:21.540200 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.540076    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c593963e-f5d8-4e09-8292-f6474dd368ac-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-m7l5h\" (UID: \"c593963e-f5d8-4e09-8292-f6474dd368ac\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"
Jul 28 18:07:21.540200 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.540148    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67gj8\" (UniqueName: \"kubernetes.io/projected/c593963e-f5d8-4e09-8292-f6474dd368ac-kube-api-access-67gj8\") pod \"isvc-predictive-xgboost-predictor-695f978978-m7l5h\" (UID: \"c593963e-f5d8-4e09-8292-f6474dd368ac\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"
Jul 28 18:07:21.540200 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.540183    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c593963e-f5d8-4e09-8292-f6474dd368ac-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-m7l5h\" (UID: \"c593963e-f5d8-4e09-8292-f6474dd368ac\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"
Jul 28 18:07:21.641297 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.641255    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c593963e-f5d8-4e09-8292-f6474dd368ac-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-m7l5h\" (UID: \"c593963e-f5d8-4e09-8292-f6474dd368ac\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"
Jul 28 18:07:21.641297 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.641301    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c593963e-f5d8-4e09-8292-f6474dd368ac-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-m7l5h\" (UID: \"c593963e-f5d8-4e09-8292-f6474dd368ac\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"
Jul 28 18:07:21.641666 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.641352    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c593963e-f5d8-4e09-8292-f6474dd368ac-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-m7l5h\" (UID: \"c593963e-f5d8-4e09-8292-f6474dd368ac\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"
Jul 28 18:07:21.641666 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.641404    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67gj8\" (UniqueName: \"kubernetes.io/projected/c593963e-f5d8-4e09-8292-f6474dd368ac-kube-api-access-67gj8\") pod \"isvc-predictive-xgboost-predictor-695f978978-m7l5h\" (UID: \"c593963e-f5d8-4e09-8292-f6474dd368ac\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"
Jul 28 18:07:21.641666 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.641438    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c593963e-f5d8-4e09-8292-f6474dd368ac-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-m7l5h\" (UID: \"c593963e-f5d8-4e09-8292-f6474dd368ac\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"
Jul 28 18:07:21.641666 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:07:21.641545    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-serving-cert: secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Jul 28 18:07:21.641666 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:07:21.641627    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c593963e-f5d8-4e09-8292-f6474dd368ac-proxy-tls podName:c593963e-f5d8-4e09-8292-f6474dd368ac nodeName:}" failed. No retries permitted until 2026-07-28 18:07:22.141606524 +0000 UTC m=+3490.416222955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c593963e-f5d8-4e09-8292-f6474dd368ac-proxy-tls") pod "isvc-predictive-xgboost-predictor-695f978978-m7l5h" (UID: "c593963e-f5d8-4e09-8292-f6474dd368ac") : secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Jul 28 18:07:21.641924 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.641791    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c593963e-f5d8-4e09-8292-f6474dd368ac-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-m7l5h\" (UID: \"c593963e-f5d8-4e09-8292-f6474dd368ac\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"
Jul 28 18:07:21.642044 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.642022    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c593963e-f5d8-4e09-8292-f6474dd368ac-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-m7l5h\" (UID: \"c593963e-f5d8-4e09-8292-f6474dd368ac\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"
Jul 28 18:07:21.642204 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.642188    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c593963e-f5d8-4e09-8292-f6474dd368ac-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-m7l5h\" (UID: \"c593963e-f5d8-4e09-8292-f6474dd368ac\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"
Jul 28 18:07:21.661359 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.661323    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67gj8\" (UniqueName: \"kubernetes.io/projected/c593963e-f5d8-4e09-8292-f6474dd368ac-kube-api-access-67gj8\") pod \"isvc-predictive-xgboost-predictor-695f978978-m7l5h\" (UID: \"c593963e-f5d8-4e09-8292-f6474dd368ac\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"
Jul 28 18:07:21.893573 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.893480    2581 generic.go:356] "Generic (PLEG): container finished" podID="f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" containerID="9ab84b0e1104c6eceddda384c1d2145b88278d52114c3c32b4498f54a34cb69b" exitCode=0
Jul 28 18:07:21.893573 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.893535    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"
Jul 28 18:07:21.893573 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.893557    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h" event={"ID":"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a","Type":"ContainerDied","Data":"9ab84b0e1104c6eceddda384c1d2145b88278d52114c3c32b4498f54a34cb69b"}
Jul 28 18:07:21.893815 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.893628    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h" event={"ID":"f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a","Type":"ContainerDied","Data":"2be030cbb3dfbec782b94106554e408e30a243bb95795ca26f132d356e8485b6"}
Jul 28 18:07:21.893815 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.893666    2581 scope.go:117] "RemoveContainer" containerID="09f107141adbec11ab4a0b1e185e5e4bf20cae36cacb86181ebdd593acbb2a44"
Jul 28 18:07:21.902834 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.902815    2581 scope.go:117] "RemoveContainer" containerID="9ab84b0e1104c6eceddda384c1d2145b88278d52114c3c32b4498f54a34cb69b"
Jul 28 18:07:21.910672 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.910648    2581 scope.go:117] "RemoveContainer" containerID="6fec9ed5110ba1117f548e72b84ed837af5df661f1b190bbc3dd5b0fc60243ef"
Jul 28 18:07:21.916552 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.916526    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"]
Jul 28 18:07:21.918995 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.918976    2581 scope.go:117] "RemoveContainer" containerID="09f107141adbec11ab4a0b1e185e5e4bf20cae36cacb86181ebdd593acbb2a44"
Jul 28 18:07:21.919352 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:07:21.919320    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09f107141adbec11ab4a0b1e185e5e4bf20cae36cacb86181ebdd593acbb2a44\": container with ID starting with 09f107141adbec11ab4a0b1e185e5e4bf20cae36cacb86181ebdd593acbb2a44 not found: ID does not exist" containerID="09f107141adbec11ab4a0b1e185e5e4bf20cae36cacb86181ebdd593acbb2a44"
Jul 28 18:07:21.919458 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.919361    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09f107141adbec11ab4a0b1e185e5e4bf20cae36cacb86181ebdd593acbb2a44"} err="failed to get container status \"09f107141adbec11ab4a0b1e185e5e4bf20cae36cacb86181ebdd593acbb2a44\": rpc error: code = NotFound desc = could not find container \"09f107141adbec11ab4a0b1e185e5e4bf20cae36cacb86181ebdd593acbb2a44\": container with ID starting with 09f107141adbec11ab4a0b1e185e5e4bf20cae36cacb86181ebdd593acbb2a44 not found: ID does not exist"
Jul 28 18:07:21.919458 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.919390    2581 scope.go:117] "RemoveContainer" containerID="9ab84b0e1104c6eceddda384c1d2145b88278d52114c3c32b4498f54a34cb69b"
Jul 28 18:07:21.919773 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:07:21.919749    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ab84b0e1104c6eceddda384c1d2145b88278d52114c3c32b4498f54a34cb69b\": container with ID starting with 9ab84b0e1104c6eceddda384c1d2145b88278d52114c3c32b4498f54a34cb69b not found: ID does not exist" containerID="9ab84b0e1104c6eceddda384c1d2145b88278d52114c3c32b4498f54a34cb69b"
Jul 28 18:07:21.919852 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.919786    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab84b0e1104c6eceddda384c1d2145b88278d52114c3c32b4498f54a34cb69b"} err="failed to get container status \"9ab84b0e1104c6eceddda384c1d2145b88278d52114c3c32b4498f54a34cb69b\": rpc error: code = NotFound desc = could not find container \"9ab84b0e1104c6eceddda384c1d2145b88278d52114c3c32b4498f54a34cb69b\": container with ID starting with 9ab84b0e1104c6eceddda384c1d2145b88278d52114c3c32b4498f54a34cb69b not found: ID does not exist"
Jul 28 18:07:21.919852 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.919809    2581 scope.go:117] "RemoveContainer" containerID="6fec9ed5110ba1117f548e72b84ed837af5df661f1b190bbc3dd5b0fc60243ef"
Jul 28 18:07:21.920131 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:07:21.920113    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fec9ed5110ba1117f548e72b84ed837af5df661f1b190bbc3dd5b0fc60243ef\": container with ID starting with 6fec9ed5110ba1117f548e72b84ed837af5df661f1b190bbc3dd5b0fc60243ef not found: ID does not exist" containerID="6fec9ed5110ba1117f548e72b84ed837af5df661f1b190bbc3dd5b0fc60243ef"
Jul 28 18:07:21.920201 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.920137    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fec9ed5110ba1117f548e72b84ed837af5df661f1b190bbc3dd5b0fc60243ef"} err="failed to get container status \"6fec9ed5110ba1117f548e72b84ed837af5df661f1b190bbc3dd5b0fc60243ef\": rpc error: code = NotFound desc = could not find container \"6fec9ed5110ba1117f548e72b84ed837af5df661f1b190bbc3dd5b0fc60243ef\": container with ID starting with 6fec9ed5110ba1117f548e72b84ed837af5df661f1b190bbc3dd5b0fc60243ef not found: ID does not exist"
Jul 28 18:07:21.920745 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:21.920728    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p5p2h"]
Jul 28 18:07:22.035594 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:22.035559    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:07:22.146452 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:22.146353    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c593963e-f5d8-4e09-8292-f6474dd368ac-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-m7l5h\" (UID: \"c593963e-f5d8-4e09-8292-f6474dd368ac\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"
Jul 28 18:07:22.148819 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:22.148795    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c593963e-f5d8-4e09-8292-f6474dd368ac-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-m7l5h\" (UID: \"c593963e-f5d8-4e09-8292-f6474dd368ac\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"
Jul 28 18:07:22.263597 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:22.263551    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"
Jul 28 18:07:22.290113 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:22.290081    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a" path="/var/lib/kubelet/pods/f70dc3d5-4fe9-4865-b3d5-363efa8a2d9a/volumes"
Jul 28 18:07:22.401463 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:22.401284    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"]
Jul 28 18:07:22.404208 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:07:22.404180    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc593963e_f5d8_4e09_8292_f6474dd368ac.slice/crio-822607c654936656fc0b40433d59b253b00423d800b76f8877a4b4956988031c WatchSource:0}: Error finding container 822607c654936656fc0b40433d59b253b00423d800b76f8877a4b4956988031c: Status 404 returned error can't find the container with id 822607c654936656fc0b40433d59b253b00423d800b76f8877a4b4956988031c
Jul 28 18:07:22.899360 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:22.899324    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h" event={"ID":"c593963e-f5d8-4e09-8292-f6474dd368ac","Type":"ContainerStarted","Data":"0a035d42a83ae7ada9cc0f770c2ec564376a0ceaf8f1624c39c99678bf854094"}
Jul 28 18:07:22.899564 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:22.899392    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h" event={"ID":"c593963e-f5d8-4e09-8292-f6474dd368ac","Type":"ContainerStarted","Data":"822607c654936656fc0b40433d59b253b00423d800b76f8877a4b4956988031c"}
Jul 28 18:07:26.917911 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:26.917873    2581 generic.go:356] "Generic (PLEG): container finished" podID="c593963e-f5d8-4e09-8292-f6474dd368ac" containerID="0a035d42a83ae7ada9cc0f770c2ec564376a0ceaf8f1624c39c99678bf854094" exitCode=0
Jul 28 18:07:26.918341 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:26.917925    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h" event={"ID":"c593963e-f5d8-4e09-8292-f6474dd368ac","Type":"ContainerDied","Data":"0a035d42a83ae7ada9cc0f770c2ec564376a0ceaf8f1624c39c99678bf854094"}
Jul 28 18:07:27.923750 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:27.923713    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h" event={"ID":"c593963e-f5d8-4e09-8292-f6474dd368ac","Type":"ContainerStarted","Data":"06e19644c63c67ea55b721b18e06082158de1690c0b95fa117dafc510d7297bb"}
Jul 28 18:07:27.924224 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:27.923775    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h" event={"ID":"c593963e-f5d8-4e09-8292-f6474dd368ac","Type":"ContainerStarted","Data":"e06ff9f8ea1058e78b0768de7372750d1668b56c4442200727924ed45d959d89"}
Jul 28 18:07:27.924224 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:27.924065    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"
Jul 28 18:07:27.924224 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:27.924202    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"
Jul 28 18:07:27.925427 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:27.925400    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h" podUID="c593963e-f5d8-4e09-8292-f6474dd368ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 28 18:07:27.949910 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:27.949851    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h" podStartSLOduration=3.108007439 podStartE2EDuration="6.949835186s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.841827747s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:07:27.947374511 +0000 UTC m=+3496.221990943" watchObservedRunningTime="2026-07-28 18:07:27.949835186 +0000 UTC m=+3496.224451691"
Jul 28 18:07:28.927558 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:28.927515    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h" podUID="c593963e-f5d8-4e09-8292-f6474dd368ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 28 18:07:33.932176 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:33.932147    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"
Jul 28 18:07:33.932679 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:33.932648    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h" podUID="c593963e-f5d8-4e09-8292-f6474dd368ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 28 18:07:43.933366 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:43.933273    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h" podUID="c593963e-f5d8-4e09-8292-f6474dd368ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 28 18:07:53.932641 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:07:53.932590    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h" podUID="c593963e-f5d8-4e09-8292-f6474dd368ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 28 18:08:03.933635 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:03.933583    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h" podUID="c593963e-f5d8-4e09-8292-f6474dd368ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 28 18:08:13.933076 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:13.933027    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h" podUID="c593963e-f5d8-4e09-8292-f6474dd368ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 28 18:08:23.933127 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:23.933077    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h" podUID="c593963e-f5d8-4e09-8292-f6474dd368ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 28 18:08:33.933474 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:33.933420    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h" podUID="c593963e-f5d8-4e09-8292-f6474dd368ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 28 18:08:43.286142 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:43.286106    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"
Jul 28 18:08:51.480898 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:51.480865    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"]
Jul 28 18:08:51.481313 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:51.481154    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h" podUID="c593963e-f5d8-4e09-8292-f6474dd368ac" containerName="kserve-container" containerID="cri-o://e06ff9f8ea1058e78b0768de7372750d1668b56c4442200727924ed45d959d89" gracePeriod=30
Jul 28 18:08:51.481313 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:51.481154    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h" podUID="c593963e-f5d8-4e09-8292-f6474dd368ac" containerName="kube-rbac-proxy" containerID="cri-o://06e19644c63c67ea55b721b18e06082158de1690c0b95fa117dafc510d7297bb" gracePeriod=30
Jul 28 18:08:51.506488 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:51.506456    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:08:52.225270 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:52.225227    2581 generic.go:356] "Generic (PLEG): container finished" podID="c593963e-f5d8-4e09-8292-f6474dd368ac" containerID="06e19644c63c67ea55b721b18e06082158de1690c0b95fa117dafc510d7297bb" exitCode=2
Jul 28 18:08:52.225457 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:52.225301    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h" event={"ID":"c593963e-f5d8-4e09-8292-f6474dd368ac","Type":"ContainerDied","Data":"06e19644c63c67ea55b721b18e06082158de1690c0b95fa117dafc510d7297bb"}
Jul 28 18:08:53.286602 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:53.286559    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h" podUID="c593963e-f5d8-4e09-8292-f6474dd368ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jul 28 18:08:53.928712 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:53.928669    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h" podUID="c593963e-f5d8-4e09-8292-f6474dd368ac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Jul 28 18:08:56.227313 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.227285    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"
Jul 28 18:08:56.241447 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.241420    2581 generic.go:356] "Generic (PLEG): container finished" podID="c593963e-f5d8-4e09-8292-f6474dd368ac" containerID="e06ff9f8ea1058e78b0768de7372750d1668b56c4442200727924ed45d959d89" exitCode=0
Jul 28 18:08:56.241580 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.241476    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h" event={"ID":"c593963e-f5d8-4e09-8292-f6474dd368ac","Type":"ContainerDied","Data":"e06ff9f8ea1058e78b0768de7372750d1668b56c4442200727924ed45d959d89"}
Jul 28 18:08:56.241580 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.241523    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h" event={"ID":"c593963e-f5d8-4e09-8292-f6474dd368ac","Type":"ContainerDied","Data":"822607c654936656fc0b40433d59b253b00423d800b76f8877a4b4956988031c"}
Jul 28 18:08:56.241580 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.241548    2581 scope.go:117] "RemoveContainer" containerID="06e19644c63c67ea55b721b18e06082158de1690c0b95fa117dafc510d7297bb"
Jul 28 18:08:56.241580 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.241478    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"
Jul 28 18:08:56.249177 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.249157    2581 scope.go:117] "RemoveContainer" containerID="e06ff9f8ea1058e78b0768de7372750d1668b56c4442200727924ed45d959d89"
Jul 28 18:08:56.257762 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.257739    2581 scope.go:117] "RemoveContainer" containerID="0a035d42a83ae7ada9cc0f770c2ec564376a0ceaf8f1624c39c99678bf854094"
Jul 28 18:08:56.264854 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.264832    2581 scope.go:117] "RemoveContainer" containerID="06e19644c63c67ea55b721b18e06082158de1690c0b95fa117dafc510d7297bb"
Jul 28 18:08:56.265135 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:08:56.265115    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06e19644c63c67ea55b721b18e06082158de1690c0b95fa117dafc510d7297bb\": container with ID starting with 06e19644c63c67ea55b721b18e06082158de1690c0b95fa117dafc510d7297bb not found: ID does not exist" containerID="06e19644c63c67ea55b721b18e06082158de1690c0b95fa117dafc510d7297bb"
Jul 28 18:08:56.265220 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.265148    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06e19644c63c67ea55b721b18e06082158de1690c0b95fa117dafc510d7297bb"} err="failed to get container status \"06e19644c63c67ea55b721b18e06082158de1690c0b95fa117dafc510d7297bb\": rpc error: code = NotFound desc = could not find container \"06e19644c63c67ea55b721b18e06082158de1690c0b95fa117dafc510d7297bb\": container with ID starting with 06e19644c63c67ea55b721b18e06082158de1690c0b95fa117dafc510d7297bb not found: ID does not exist"
Jul 28 18:08:56.265220 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.265174    2581 scope.go:117] "RemoveContainer" containerID="e06ff9f8ea1058e78b0768de7372750d1668b56c4442200727924ed45d959d89"
Jul 28 18:08:56.265423 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:08:56.265407    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e06ff9f8ea1058e78b0768de7372750d1668b56c4442200727924ed45d959d89\": container with ID starting with e06ff9f8ea1058e78b0768de7372750d1668b56c4442200727924ed45d959d89 not found: ID does not exist" containerID="e06ff9f8ea1058e78b0768de7372750d1668b56c4442200727924ed45d959d89"
Jul 28 18:08:56.265496 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.265430    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e06ff9f8ea1058e78b0768de7372750d1668b56c4442200727924ed45d959d89"} err="failed to get container status \"e06ff9f8ea1058e78b0768de7372750d1668b56c4442200727924ed45d959d89\": rpc error: code = NotFound desc = could not find container \"e06ff9f8ea1058e78b0768de7372750d1668b56c4442200727924ed45d959d89\": container with ID starting with e06ff9f8ea1058e78b0768de7372750d1668b56c4442200727924ed45d959d89 not found: ID does not exist"
Jul 28 18:08:56.265496 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.265450    2581 scope.go:117] "RemoveContainer" containerID="0a035d42a83ae7ada9cc0f770c2ec564376a0ceaf8f1624c39c99678bf854094"
Jul 28 18:08:56.265672 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:08:56.265656    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a035d42a83ae7ada9cc0f770c2ec564376a0ceaf8f1624c39c99678bf854094\": container with ID starting with 0a035d42a83ae7ada9cc0f770c2ec564376a0ceaf8f1624c39c99678bf854094 not found: ID does not exist" containerID="0a035d42a83ae7ada9cc0f770c2ec564376a0ceaf8f1624c39c99678bf854094"
Jul 28 18:08:56.265727 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.265680    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a035d42a83ae7ada9cc0f770c2ec564376a0ceaf8f1624c39c99678bf854094"} err="failed to get container status \"0a035d42a83ae7ada9cc0f770c2ec564376a0ceaf8f1624c39c99678bf854094\": rpc error: code = NotFound desc = could not find container \"0a035d42a83ae7ada9cc0f770c2ec564376a0ceaf8f1624c39c99678bf854094\": container with ID starting with 0a035d42a83ae7ada9cc0f770c2ec564376a0ceaf8f1624c39c99678bf854094 not found: ID does not exist"
Jul 28 18:08:56.271929 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.271910    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c593963e-f5d8-4e09-8292-f6474dd368ac-kube-rbac-proxy-sar-config\") pod \"c593963e-f5d8-4e09-8292-f6474dd368ac\" (UID: \"c593963e-f5d8-4e09-8292-f6474dd368ac\") "
Jul 28 18:08:56.272033 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.271936    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c593963e-f5d8-4e09-8292-f6474dd368ac-proxy-tls\") pod \"c593963e-f5d8-4e09-8292-f6474dd368ac\" (UID: \"c593963e-f5d8-4e09-8292-f6474dd368ac\") "
Jul 28 18:08:56.272093 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.272066    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c593963e-f5d8-4e09-8292-f6474dd368ac-cabundle-cert\") pod \"c593963e-f5d8-4e09-8292-f6474dd368ac\" (UID: \"c593963e-f5d8-4e09-8292-f6474dd368ac\") "
Jul 28 18:08:56.272144 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.272092    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67gj8\" (UniqueName: \"kubernetes.io/projected/c593963e-f5d8-4e09-8292-f6474dd368ac-kube-api-access-67gj8\") pod \"c593963e-f5d8-4e09-8292-f6474dd368ac\" (UID: \"c593963e-f5d8-4e09-8292-f6474dd368ac\") "
Jul 28 18:08:56.272144 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.272122    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c593963e-f5d8-4e09-8292-f6474dd368ac-kserve-provision-location\") pod \"c593963e-f5d8-4e09-8292-f6474dd368ac\" (UID: \"c593963e-f5d8-4e09-8292-f6474dd368ac\") "
Jul 28 18:08:56.272394 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.272356    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c593963e-f5d8-4e09-8292-f6474dd368ac-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c593963e-f5d8-4e09-8292-f6474dd368ac" (UID: "c593963e-f5d8-4e09-8292-f6474dd368ac"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:08:56.272484 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.272417    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c593963e-f5d8-4e09-8292-f6474dd368ac-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c593963e-f5d8-4e09-8292-f6474dd368ac" (UID: "c593963e-f5d8-4e09-8292-f6474dd368ac"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:08:56.272535 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.272492    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c593963e-f5d8-4e09-8292-f6474dd368ac-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c593963e-f5d8-4e09-8292-f6474dd368ac" (UID: "c593963e-f5d8-4e09-8292-f6474dd368ac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:08:56.274082 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.274058    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c593963e-f5d8-4e09-8292-f6474dd368ac-kube-api-access-67gj8" (OuterVolumeSpecName: "kube-api-access-67gj8") pod "c593963e-f5d8-4e09-8292-f6474dd368ac" (UID: "c593963e-f5d8-4e09-8292-f6474dd368ac"). InnerVolumeSpecName "kube-api-access-67gj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:08:56.274206 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.274190    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c593963e-f5d8-4e09-8292-f6474dd368ac-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c593963e-f5d8-4e09-8292-f6474dd368ac" (UID: "c593963e-f5d8-4e09-8292-f6474dd368ac"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:08:56.373216 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.373121    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67gj8\" (UniqueName: \"kubernetes.io/projected/c593963e-f5d8-4e09-8292-f6474dd368ac-kube-api-access-67gj8\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:08:56.373216 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.373156    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c593963e-f5d8-4e09-8292-f6474dd368ac-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:08:56.373216 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.373166    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c593963e-f5d8-4e09-8292-f6474dd368ac-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:08:56.373216 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.373175    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c593963e-f5d8-4e09-8292-f6474dd368ac-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:08:56.373216 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.373185    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c593963e-f5d8-4e09-8292-f6474dd368ac-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:08:56.562904 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.562871    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"]
Jul 28 18:08:56.566316 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.566292    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-m7l5h"]
Jul 28 18:08:56.586468 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.586440    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:08:56.661434 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.661399    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"]
Jul 28 18:08:56.661647 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.661635    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c593963e-f5d8-4e09-8292-f6474dd368ac" containerName="kserve-container"
Jul 28 18:08:56.661647 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.661648    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c593963e-f5d8-4e09-8292-f6474dd368ac" containerName="kserve-container"
Jul 28 18:08:56.661763 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.661670    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c593963e-f5d8-4e09-8292-f6474dd368ac" containerName="storage-initializer"
Jul 28 18:08:56.661763 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.661676    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c593963e-f5d8-4e09-8292-f6474dd368ac" containerName="storage-initializer"
Jul 28 18:08:56.661763 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.661684    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c593963e-f5d8-4e09-8292-f6474dd368ac" containerName="kube-rbac-proxy"
Jul 28 18:08:56.661763 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.661689    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c593963e-f5d8-4e09-8292-f6474dd368ac" containerName="kube-rbac-proxy"
Jul 28 18:08:56.661763 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.661728    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c593963e-f5d8-4e09-8292-f6474dd368ac" containerName="kube-rbac-proxy"
Jul 28 18:08:56.661763 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.661736    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c593963e-f5d8-4e09-8292-f6474dd368ac" containerName="kserve-container"
Jul 28 18:08:56.664645 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.664626    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"
Jul 28 18:08:56.667364 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.667345    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Jul 28 18:08:56.669077 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.669061    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:08:56.669478 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.669460    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:08:56.669662 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.669554    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Jul 28 18:08:56.669662 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.669559    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 18:08:56.669804 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.669790    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:08:56.669924 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.669907    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:08:56.676984 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.676935    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"]
Jul 28 18:08:56.776106 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.776071    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d520360a-3ca3-4e45-8be8-074cc168bf57-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz\" (UID: \"d520360a-3ca3-4e45-8be8-074cc168bf57\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"
Jul 28 18:08:56.776106 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.776114    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d520360a-3ca3-4e45-8be8-074cc168bf57-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz\" (UID: \"d520360a-3ca3-4e45-8be8-074cc168bf57\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"
Jul 28 18:08:56.776351 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.776134    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkr24\" (UniqueName: \"kubernetes.io/projected/d520360a-3ca3-4e45-8be8-074cc168bf57-kube-api-access-zkr24\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz\" (UID: \"d520360a-3ca3-4e45-8be8-074cc168bf57\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"
Jul 28 18:08:56.776351 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.776177    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d520360a-3ca3-4e45-8be8-074cc168bf57-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz\" (UID: \"d520360a-3ca3-4e45-8be8-074cc168bf57\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"
Jul 28 18:08:56.776351 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.776211    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d520360a-3ca3-4e45-8be8-074cc168bf57-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz\" (UID: \"d520360a-3ca3-4e45-8be8-074cc168bf57\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"
Jul 28 18:08:56.876696 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.876653    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d520360a-3ca3-4e45-8be8-074cc168bf57-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz\" (UID: \"d520360a-3ca3-4e45-8be8-074cc168bf57\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"
Jul 28 18:08:56.876696 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.876692    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d520360a-3ca3-4e45-8be8-074cc168bf57-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz\" (UID: \"d520360a-3ca3-4e45-8be8-074cc168bf57\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"
Jul 28 18:08:56.876975 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.876736    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d520360a-3ca3-4e45-8be8-074cc168bf57-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz\" (UID: \"d520360a-3ca3-4e45-8be8-074cc168bf57\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"
Jul 28 18:08:56.876975 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.876757    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d520360a-3ca3-4e45-8be8-074cc168bf57-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz\" (UID: \"d520360a-3ca3-4e45-8be8-074cc168bf57\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"
Jul 28 18:08:56.876975 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.876776    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkr24\" (UniqueName: \"kubernetes.io/projected/d520360a-3ca3-4e45-8be8-074cc168bf57-kube-api-access-zkr24\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz\" (UID: \"d520360a-3ca3-4e45-8be8-074cc168bf57\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"
Jul 28 18:08:56.877207 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.877184    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d520360a-3ca3-4e45-8be8-074cc168bf57-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz\" (UID: \"d520360a-3ca3-4e45-8be8-074cc168bf57\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"
Jul 28 18:08:56.877452 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.877431    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d520360a-3ca3-4e45-8be8-074cc168bf57-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz\" (UID: \"d520360a-3ca3-4e45-8be8-074cc168bf57\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"
Jul 28 18:08:56.877536 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.877506    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d520360a-3ca3-4e45-8be8-074cc168bf57-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz\" (UID: \"d520360a-3ca3-4e45-8be8-074cc168bf57\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"
Jul 28 18:08:56.879462 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.879443    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d520360a-3ca3-4e45-8be8-074cc168bf57-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz\" (UID: \"d520360a-3ca3-4e45-8be8-074cc168bf57\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"
Jul 28 18:08:56.889278 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.889255    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkr24\" (UniqueName: \"kubernetes.io/projected/d520360a-3ca3-4e45-8be8-074cc168bf57-kube-api-access-zkr24\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz\" (UID: \"d520360a-3ca3-4e45-8be8-074cc168bf57\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"
Jul 28 18:08:56.923643 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.923569    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:08:56.974434 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:56.974393    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"
Jul 28 18:08:57.109197 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:57.109149    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"]
Jul 28 18:08:57.111706 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:08:57.111675    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd520360a_3ca3_4e45_8be8_074cc168bf57.slice/crio-081c57923eff32dcac6f9459709ada246841d5814ffd1402ce0f740465a015e3 WatchSource:0}: Error finding container 081c57923eff32dcac6f9459709ada246841d5814ffd1402ce0f740465a015e3: Status 404 returned error can't find the container with id 081c57923eff32dcac6f9459709ada246841d5814ffd1402ce0f740465a015e3
Jul 28 18:08:57.247195 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:57.247095    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz" event={"ID":"d520360a-3ca3-4e45-8be8-074cc168bf57","Type":"ContainerStarted","Data":"efe30858db4020bbfb8deadc65bb4473549aeb02667b9a7d897f915ab2a92a3e"}
Jul 28 18:08:57.247195 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:57.247162    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz" event={"ID":"d520360a-3ca3-4e45-8be8-074cc168bf57","Type":"ContainerStarted","Data":"081c57923eff32dcac6f9459709ada246841d5814ffd1402ce0f740465a015e3"}
Jul 28 18:08:58.289711 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:08:58.289675    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c593963e-f5d8-4e09-8292-f6474dd368ac" path="/var/lib/kubelet/pods/c593963e-f5d8-4e09-8292-f6474dd368ac/volumes"
Jul 28 18:09:01.268216 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:09:01.268116    2581 generic.go:356] "Generic (PLEG): container finished" podID="d520360a-3ca3-4e45-8be8-074cc168bf57" containerID="efe30858db4020bbfb8deadc65bb4473549aeb02667b9a7d897f915ab2a92a3e" exitCode=0
Jul 28 18:09:01.268216 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:09:01.268183    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz" event={"ID":"d520360a-3ca3-4e45-8be8-074cc168bf57","Type":"ContainerDied","Data":"efe30858db4020bbfb8deadc65bb4473549aeb02667b9a7d897f915ab2a92a3e"}
Jul 28 18:09:02.274108 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:09:02.274078    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz" event={"ID":"d520360a-3ca3-4e45-8be8-074cc168bf57","Type":"ContainerStarted","Data":"05ce47bcf4e4c37d96a85a6417d84f5da624e0c7a8de0f56595f8d5d2f176190"}
Jul 28 18:09:02.274514 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:09:02.274137    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz" event={"ID":"d520360a-3ca3-4e45-8be8-074cc168bf57","Type":"ContainerStarted","Data":"635c56422f13e53be7600363ba00cd2f9b65a60d82e198675a532b8fd0246e69"}
Jul 28 18:09:02.274514 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:09:02.274398    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"
Jul 28 18:09:02.274661 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:09:02.274547    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"
Jul 28 18:09:02.275541 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:09:02.275514    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz" podUID="d520360a-3ca3-4e45-8be8-074cc168bf57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 18:09:02.300575 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:09:02.300524    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz" podStartSLOduration=2.463489935 podStartE2EDuration="6.300508217s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.837018282s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:09:02.299117487 +0000 UTC m=+3590.573733918" watchObservedRunningTime="2026-07-28 18:09:02.300508217 +0000 UTC m=+3590.575124650"
Jul 28 18:09:03.277403 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:09:03.277367    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz" podUID="d520360a-3ca3-4e45-8be8-074cc168bf57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 18:09:08.281695 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:09:08.281624    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"
Jul 28 18:09:08.282156 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:09:08.282126    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz" podUID="d520360a-3ca3-4e45-8be8-074cc168bf57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 18:09:12.399703 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:09:12.399667    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 18:09:12.403567 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:09:12.403548    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 18:09:12.592061 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:09:12.592033    2581 scope.go:117] "RemoveContainer" containerID="85bb2b2cadee3421f318c4ab59a1d893cdaeb39e4157d842499acb9529c99be2"
Jul 28 18:09:12.599487 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:09:12.599467    2581 scope.go:117] "RemoveContainer" containerID="c79b453874b7dc80def92dbbf2825963777819719886720654529a3142f14460"
Jul 28 18:09:12.605984 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:09:12.605966    2581 scope.go:117] "RemoveContainer" containerID="fe001861c5d670f9273ba280e4de5c62aca0bae3d471d9b8726393d4572c7fd2"
Jul 28 18:09:18.282396 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:09:18.282349    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz" podUID="d520360a-3ca3-4e45-8be8-074cc168bf57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 18:09:28.282162 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:09:28.282123    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz" podUID="d520360a-3ca3-4e45-8be8-074cc168bf57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 18:09:38.283171 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:09:38.283125    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz" podUID="d520360a-3ca3-4e45-8be8-074cc168bf57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 18:09:48.282772 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:09:48.282729    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz" podUID="d520360a-3ca3-4e45-8be8-074cc168bf57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 18:09:58.282784 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:09:58.282744    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz" podUID="d520360a-3ca3-4e45-8be8-074cc168bf57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 18:10:08.282827 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:08.282784    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz" podUID="d520360a-3ca3-4e45-8be8-074cc168bf57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 18:10:18.283099 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:18.283069    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"
Jul 28 18:10:26.807371 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:26.807331    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"]
Jul 28 18:10:26.808005 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:26.807580    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz" podUID="d520360a-3ca3-4e45-8be8-074cc168bf57" containerName="kserve-container" containerID="cri-o://635c56422f13e53be7600363ba00cd2f9b65a60d82e198675a532b8fd0246e69" gracePeriod=30
Jul 28 18:10:26.808005 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:26.807639    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz" podUID="d520360a-3ca3-4e45-8be8-074cc168bf57" containerName="kube-rbac-proxy" containerID="cri-o://05ce47bcf4e4c37d96a85a6417d84f5da624e0c7a8de0f56595f8d5d2f176190" gracePeriod=30
Jul 28 18:10:26.827535 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:26.827501    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:10:27.569197 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:27.569161    2581 generic.go:356] "Generic (PLEG): container finished" podID="d520360a-3ca3-4e45-8be8-074cc168bf57" containerID="05ce47bcf4e4c37d96a85a6417d84f5da624e0c7a8de0f56595f8d5d2f176190" exitCode=2
Jul 28 18:10:27.569370 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:27.569231    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz" event={"ID":"d520360a-3ca3-4e45-8be8-074cc168bf57","Type":"ContainerDied","Data":"05ce47bcf4e4c37d96a85a6417d84f5da624e0c7a8de0f56595f8d5d2f176190"}
Jul 28 18:10:28.278324 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:28.278273    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz" podUID="d520360a-3ca3-4e45-8be8-074cc168bf57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.51:8643/healthz\": dial tcp 10.133.0.51:8643: connect: connection refused"
Jul 28 18:10:28.282588 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:28.282561    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz" podUID="d520360a-3ca3-4e45-8be8-074cc168bf57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 28 18:10:31.585321 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:31.585286    2581 generic.go:356] "Generic (PLEG): container finished" podID="d520360a-3ca3-4e45-8be8-074cc168bf57" containerID="635c56422f13e53be7600363ba00cd2f9b65a60d82e198675a532b8fd0246e69" exitCode=0
Jul 28 18:10:31.585706 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:31.585358    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz" event={"ID":"d520360a-3ca3-4e45-8be8-074cc168bf57","Type":"ContainerDied","Data":"635c56422f13e53be7600363ba00cd2f9b65a60d82e198675a532b8fd0246e69"}
Jul 28 18:10:31.970822 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:31.970782    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:10:32.057839 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.057806    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"]
Jul 28 18:10:32.061390 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.061369    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"
Jul 28 18:10:32.062135 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.062116    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"
Jul 28 18:10:32.064473 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.064453    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Jul 28 18:10:32.064584 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.064531    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 28 18:10:32.081302 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.081268    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"]
Jul 28 18:10:32.155984 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.155925    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw\" (UID: \"5e4e10cb-8be6-4573-bcb3-f3abbe356f79\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"
Jul 28 18:10:32.156160 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.156020    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw\" (UID: \"5e4e10cb-8be6-4573-bcb3-f3abbe356f79\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"
Jul 28 18:10:32.156160 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.156059    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw\" (UID: \"5e4e10cb-8be6-4573-bcb3-f3abbe356f79\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"
Jul 28 18:10:32.156160 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.156079    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8x76\" (UniqueName: \"kubernetes.io/projected/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-kube-api-access-w8x76\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw\" (UID: \"5e4e10cb-8be6-4573-bcb3-f3abbe356f79\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"
Jul 28 18:10:32.156160 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.156099    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw\" (UID: \"5e4e10cb-8be6-4573-bcb3-f3abbe356f79\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"
Jul 28 18:10:32.256723 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.256621    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d520360a-3ca3-4e45-8be8-074cc168bf57-kserve-provision-location\") pod \"d520360a-3ca3-4e45-8be8-074cc168bf57\" (UID: \"d520360a-3ca3-4e45-8be8-074cc168bf57\") "
Jul 28 18:10:32.256723 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.256665    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d520360a-3ca3-4e45-8be8-074cc168bf57-cabundle-cert\") pod \"d520360a-3ca3-4e45-8be8-074cc168bf57\" (UID: \"d520360a-3ca3-4e45-8be8-074cc168bf57\") "
Jul 28 18:10:32.256723 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.256698    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d520360a-3ca3-4e45-8be8-074cc168bf57-kube-rbac-proxy-sar-config\") pod \"d520360a-3ca3-4e45-8be8-074cc168bf57\" (UID: \"d520360a-3ca3-4e45-8be8-074cc168bf57\") "
Jul 28 18:10:32.257170 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.256732    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d520360a-3ca3-4e45-8be8-074cc168bf57-proxy-tls\") pod \"d520360a-3ca3-4e45-8be8-074cc168bf57\" (UID: \"d520360a-3ca3-4e45-8be8-074cc168bf57\") "
Jul 28 18:10:32.257170 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.256767    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkr24\" (UniqueName: \"kubernetes.io/projected/d520360a-3ca3-4e45-8be8-074cc168bf57-kube-api-access-zkr24\") pod \"d520360a-3ca3-4e45-8be8-074cc168bf57\" (UID: \"d520360a-3ca3-4e45-8be8-074cc168bf57\") "
Jul 28 18:10:32.257170 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.256917    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw\" (UID: \"5e4e10cb-8be6-4573-bcb3-f3abbe356f79\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"
Jul 28 18:10:32.257170 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.257000    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8x76\" (UniqueName: \"kubernetes.io/projected/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-kube-api-access-w8x76\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw\" (UID: \"5e4e10cb-8be6-4573-bcb3-f3abbe356f79\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"
Jul 28 18:10:32.257170 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.257037    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw\" (UID: \"5e4e10cb-8be6-4573-bcb3-f3abbe356f79\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"
Jul 28 18:10:32.257170 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.257102    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d520360a-3ca3-4e45-8be8-074cc168bf57-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d520360a-3ca3-4e45-8be8-074cc168bf57" (UID: "d520360a-3ca3-4e45-8be8-074cc168bf57"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:10:32.257501 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.257177    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw\" (UID: \"5e4e10cb-8be6-4573-bcb3-f3abbe356f79\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"
Jul 28 18:10:32.257501 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.257203    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d520360a-3ca3-4e45-8be8-074cc168bf57-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d520360a-3ca3-4e45-8be8-074cc168bf57" (UID: "d520360a-3ca3-4e45-8be8-074cc168bf57"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:10:32.257501 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.257215    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw\" (UID: \"5e4e10cb-8be6-4573-bcb3-f3abbe356f79\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"
Jul 28 18:10:32.257501 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.257113    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d520360a-3ca3-4e45-8be8-074cc168bf57-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d520360a-3ca3-4e45-8be8-074cc168bf57" (UID: "d520360a-3ca3-4e45-8be8-074cc168bf57"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:10:32.257501 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.257363    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d520360a-3ca3-4e45-8be8-074cc168bf57-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:10:32.257501 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.257381    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d520360a-3ca3-4e45-8be8-074cc168bf57-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:10:32.257501 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.257395    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d520360a-3ca3-4e45-8be8-074cc168bf57-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:10:32.258017 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.257990    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw\" (UID: \"5e4e10cb-8be6-4573-bcb3-f3abbe356f79\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"
Jul 28 18:10:32.258158 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.258135    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw\" (UID: \"5e4e10cb-8be6-4573-bcb3-f3abbe356f79\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"
Jul 28 18:10:32.258158 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.258151    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw\" (UID: \"5e4e10cb-8be6-4573-bcb3-f3abbe356f79\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"
Jul 28 18:10:32.259179 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.259155    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d520360a-3ca3-4e45-8be8-074cc168bf57-kube-api-access-zkr24" (OuterVolumeSpecName: "kube-api-access-zkr24") pod "d520360a-3ca3-4e45-8be8-074cc168bf57" (UID: "d520360a-3ca3-4e45-8be8-074cc168bf57"). InnerVolumeSpecName "kube-api-access-zkr24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:10:32.259502 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.259480    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d520360a-3ca3-4e45-8be8-074cc168bf57-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d520360a-3ca3-4e45-8be8-074cc168bf57" (UID: "d520360a-3ca3-4e45-8be8-074cc168bf57"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:10:32.259776 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.259760    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw\" (UID: \"5e4e10cb-8be6-4573-bcb3-f3abbe356f79\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"
Jul 28 18:10:32.262051 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.262027    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:10:32.269108 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.269085    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8x76\" (UniqueName: \"kubernetes.io/projected/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-kube-api-access-w8x76\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw\" (UID: \"5e4e10cb-8be6-4573-bcb3-f3abbe356f79\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"
Jul 28 18:10:32.357833 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.357798    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkr24\" (UniqueName: \"kubernetes.io/projected/d520360a-3ca3-4e45-8be8-074cc168bf57-kube-api-access-zkr24\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:10:32.357833 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.357825    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d520360a-3ca3-4e45-8be8-074cc168bf57-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:10:32.371778 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.371743    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"
Jul 28 18:10:32.509834 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.509748    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"]
Jul 28 18:10:32.513042 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:10:32.513012    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e4e10cb_8be6_4573_bcb3_f3abbe356f79.slice/crio-f4930d52a149a67c7ffb745d16d3ac867b42c33bffa68699b8b101ec3e027d6e WatchSource:0}: Error finding container f4930d52a149a67c7ffb745d16d3ac867b42c33bffa68699b8b101ec3e027d6e: Status 404 returned error can't find the container with id f4930d52a149a67c7ffb745d16d3ac867b42c33bffa68699b8b101ec3e027d6e
Jul 28 18:10:32.593515 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.593479    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"
Jul 28 18:10:32.594000 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.593485    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz" event={"ID":"d520360a-3ca3-4e45-8be8-074cc168bf57","Type":"ContainerDied","Data":"081c57923eff32dcac6f9459709ada246841d5814ffd1402ce0f740465a015e3"}
Jul 28 18:10:32.594000 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.593628    2581 scope.go:117] "RemoveContainer" containerID="05ce47bcf4e4c37d96a85a6417d84f5da624e0c7a8de0f56595f8d5d2f176190"
Jul 28 18:10:32.595615 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.595505    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw" event={"ID":"5e4e10cb-8be6-4573-bcb3-f3abbe356f79","Type":"ContainerStarted","Data":"952e05b522ad19383d982b59618e5435257e80a7a61ed277b6bfedcbabd15b3a"}
Jul 28 18:10:32.595615 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.595566    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw" event={"ID":"5e4e10cb-8be6-4573-bcb3-f3abbe356f79","Type":"ContainerStarted","Data":"f4930d52a149a67c7ffb745d16d3ac867b42c33bffa68699b8b101ec3e027d6e"}
Jul 28 18:10:32.604191 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.604168    2581 scope.go:117] "RemoveContainer" containerID="635c56422f13e53be7600363ba00cd2f9b65a60d82e198675a532b8fd0246e69"
Jul 28 18:10:32.613319 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.613291    2581 scope.go:117] "RemoveContainer" containerID="efe30858db4020bbfb8deadc65bb4473549aeb02667b9a7d897f915ab2a92a3e"
Jul 28 18:10:32.620039 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.620006    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"]
Jul 28 18:10:32.623227 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:32.623201    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-8z4nz"]
Jul 28 18:10:34.289441 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:34.289407    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d520360a-3ca3-4e45-8be8-074cc168bf57" path="/var/lib/kubelet/pods/d520360a-3ca3-4e45-8be8-074cc168bf57/volumes"
Jul 28 18:10:36.612657 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:36.612629    2581 generic.go:356] "Generic (PLEG): container finished" podID="5e4e10cb-8be6-4573-bcb3-f3abbe356f79" containerID="952e05b522ad19383d982b59618e5435257e80a7a61ed277b6bfedcbabd15b3a" exitCode=0
Jul 28 18:10:36.612964 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:36.612704    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw" event={"ID":"5e4e10cb-8be6-4573-bcb3-f3abbe356f79","Type":"ContainerDied","Data":"952e05b522ad19383d982b59618e5435257e80a7a61ed277b6bfedcbabd15b3a"}
Jul 28 18:10:37.618758 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:37.618721    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw" event={"ID":"5e4e10cb-8be6-4573-bcb3-f3abbe356f79","Type":"ContainerStarted","Data":"3dc39877fefc7b8e7fac427195f93e079162352f5b371dd09c5234ec1223319b"}
Jul 28 18:10:37.619182 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:37.618784    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw" event={"ID":"5e4e10cb-8be6-4573-bcb3-f3abbe356f79","Type":"ContainerStarted","Data":"e8e0992b354db71a5769654d9ed76b1c5c5ea34eaf317624f8e5f17370765cab"}
Jul 28 18:10:37.619182 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:37.618988    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"
Jul 28 18:10:37.619182 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:37.619037    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"
Jul 28 18:10:37.642260 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:37.642207    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw" podStartSLOduration=1.615578709 podStartE2EDuration="5.642188391s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.026609682s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:10:37.64100128 +0000 UTC m=+3685.915617712" watchObservedRunningTime="2026-07-28 18:10:37.642188391 +0000 UTC m=+3685.916804824"
Jul 28 18:10:43.628408 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:10:43.628376    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"
Jul 28 18:11:13.629378 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:11:13.629336    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw" podUID="5e4e10cb-8be6-4573-bcb3-f3abbe356f79" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.52:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 28 18:11:23.629038 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:11:23.628989    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw" podUID="5e4e10cb-8be6-4573-bcb3-f3abbe356f79" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.52:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 28 18:11:24.816822 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:11:24.816785    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:11:33.629306 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:11:33.629266    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw" podUID="5e4e10cb-8be6-4573-bcb3-f3abbe356f79" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.52:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 28 18:11:43.629870 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:11:43.629829    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw" podUID="5e4e10cb-8be6-4573-bcb3-f3abbe356f79" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.52:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 28 18:11:50.531514 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:11:50.531475    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:11:53.663368 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:11:53.663336    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"
Jul 28 18:12:03.006495 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:03.006451    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"]
Jul 28 18:12:03.006968 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:03.006703    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw" podUID="5e4e10cb-8be6-4573-bcb3-f3abbe356f79" containerName="kserve-container" containerID="cri-o://e8e0992b354db71a5769654d9ed76b1c5c5ea34eaf317624f8e5f17370765cab" gracePeriod=30
Jul 28 18:12:03.006968 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:03.006747    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw" podUID="5e4e10cb-8be6-4573-bcb3-f3abbe356f79" containerName="kube-rbac-proxy" containerID="cri-o://3dc39877fefc7b8e7fac427195f93e079162352f5b371dd09c5234ec1223319b" gracePeriod=30
Jul 28 18:12:03.007529 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:03.007505    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:12:03.623792 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:03.623741    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw" podUID="5e4e10cb-8be6-4573-bcb3-f3abbe356f79" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Jul 28 18:12:03.629205 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:03.629173    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw" podUID="5e4e10cb-8be6-4573-bcb3-f3abbe356f79" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.52:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 28 18:12:03.927841 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:03.927807    2581 generic.go:356] "Generic (PLEG): container finished" podID="5e4e10cb-8be6-4573-bcb3-f3abbe356f79" containerID="3dc39877fefc7b8e7fac427195f93e079162352f5b371dd09c5234ec1223319b" exitCode=2
Jul 28 18:12:03.928042 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:03.927879    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw" event={"ID":"5e4e10cb-8be6-4573-bcb3-f3abbe356f79","Type":"ContainerDied","Data":"3dc39877fefc7b8e7fac427195f93e079162352f5b371dd09c5234ec1223319b"}
Jul 28 18:12:07.752329 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.752298    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"
Jul 28 18:12:07.761414 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.761393    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-proxy-tls\") pod \"5e4e10cb-8be6-4573-bcb3-f3abbe356f79\" (UID: \"5e4e10cb-8be6-4573-bcb3-f3abbe356f79\") "
Jul 28 18:12:07.761508 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.761430    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-kserve-provision-location\") pod \"5e4e10cb-8be6-4573-bcb3-f3abbe356f79\" (UID: \"5e4e10cb-8be6-4573-bcb3-f3abbe356f79\") "
Jul 28 18:12:07.761508 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.761450    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-cabundle-cert\") pod \"5e4e10cb-8be6-4573-bcb3-f3abbe356f79\" (UID: \"5e4e10cb-8be6-4573-bcb3-f3abbe356f79\") "
Jul 28 18:12:07.761508 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.761467    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8x76\" (UniqueName: \"kubernetes.io/projected/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-kube-api-access-w8x76\") pod \"5e4e10cb-8be6-4573-bcb3-f3abbe356f79\" (UID: \"5e4e10cb-8be6-4573-bcb3-f3abbe356f79\") "
Jul 28 18:12:07.761508 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.761490    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-kube-rbac-proxy-sar-config\") pod \"5e4e10cb-8be6-4573-bcb3-f3abbe356f79\" (UID: \"5e4e10cb-8be6-4573-bcb3-f3abbe356f79\") "
Jul 28 18:12:07.761822 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.761792    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5e4e10cb-8be6-4573-bcb3-f3abbe356f79" (UID: "5e4e10cb-8be6-4573-bcb3-f3abbe356f79"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:12:07.761881 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.761823    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5e4e10cb-8be6-4573-bcb3-f3abbe356f79" (UID: "5e4e10cb-8be6-4573-bcb3-f3abbe356f79"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:12:07.761921 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.761885    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5e4e10cb-8be6-4573-bcb3-f3abbe356f79" (UID: "5e4e10cb-8be6-4573-bcb3-f3abbe356f79"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:12:07.763586 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.763559    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-kube-api-access-w8x76" (OuterVolumeSpecName: "kube-api-access-w8x76") pod "5e4e10cb-8be6-4573-bcb3-f3abbe356f79" (UID: "5e4e10cb-8be6-4573-bcb3-f3abbe356f79"). InnerVolumeSpecName "kube-api-access-w8x76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:12:07.763671 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.763595    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5e4e10cb-8be6-4573-bcb3-f3abbe356f79" (UID: "5e4e10cb-8be6-4573-bcb3-f3abbe356f79"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:12:07.862401 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.862307    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:12:07.862401 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.862342    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:12:07.862401 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.862353    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:12:07.862401 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.862362    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8x76\" (UniqueName: \"kubernetes.io/projected/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-kube-api-access-w8x76\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:12:07.862401 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.862370    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e4e10cb-8be6-4573-bcb3-f3abbe356f79-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:12:07.943390 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.943351    2581 generic.go:356] "Generic (PLEG): container finished" podID="5e4e10cb-8be6-4573-bcb3-f3abbe356f79" containerID="e8e0992b354db71a5769654d9ed76b1c5c5ea34eaf317624f8e5f17370765cab" exitCode=0
Jul 28 18:12:07.943564 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.943400    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"
Jul 28 18:12:07.943564 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.943398    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw" event={"ID":"5e4e10cb-8be6-4573-bcb3-f3abbe356f79","Type":"ContainerDied","Data":"e8e0992b354db71a5769654d9ed76b1c5c5ea34eaf317624f8e5f17370765cab"}
Jul 28 18:12:07.943564 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.943494    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw" event={"ID":"5e4e10cb-8be6-4573-bcb3-f3abbe356f79","Type":"ContainerDied","Data":"f4930d52a149a67c7ffb745d16d3ac867b42c33bffa68699b8b101ec3e027d6e"}
Jul 28 18:12:07.943564 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.943522    2581 scope.go:117] "RemoveContainer" containerID="3dc39877fefc7b8e7fac427195f93e079162352f5b371dd09c5234ec1223319b"
Jul 28 18:12:07.952249 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.952202    2581 scope.go:117] "RemoveContainer" containerID="e8e0992b354db71a5769654d9ed76b1c5c5ea34eaf317624f8e5f17370765cab"
Jul 28 18:12:07.959627 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.959608    2581 scope.go:117] "RemoveContainer" containerID="952e05b522ad19383d982b59618e5435257e80a7a61ed277b6bfedcbabd15b3a"
Jul 28 18:12:07.966466 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.966443    2581 scope.go:117] "RemoveContainer" containerID="3dc39877fefc7b8e7fac427195f93e079162352f5b371dd09c5234ec1223319b"
Jul 28 18:12:07.966729 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:12:07.966711    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dc39877fefc7b8e7fac427195f93e079162352f5b371dd09c5234ec1223319b\": container with ID starting with 3dc39877fefc7b8e7fac427195f93e079162352f5b371dd09c5234ec1223319b not found: ID does not exist" containerID="3dc39877fefc7b8e7fac427195f93e079162352f5b371dd09c5234ec1223319b"
Jul 28 18:12:07.966794 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.966736    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dc39877fefc7b8e7fac427195f93e079162352f5b371dd09c5234ec1223319b"} err="failed to get container status \"3dc39877fefc7b8e7fac427195f93e079162352f5b371dd09c5234ec1223319b\": rpc error: code = NotFound desc = could not find container \"3dc39877fefc7b8e7fac427195f93e079162352f5b371dd09c5234ec1223319b\": container with ID starting with 3dc39877fefc7b8e7fac427195f93e079162352f5b371dd09c5234ec1223319b not found: ID does not exist"
Jul 28 18:12:07.966794 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.966754    2581 scope.go:117] "RemoveContainer" containerID="e8e0992b354db71a5769654d9ed76b1c5c5ea34eaf317624f8e5f17370765cab"
Jul 28 18:12:07.966979 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:12:07.966963    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8e0992b354db71a5769654d9ed76b1c5c5ea34eaf317624f8e5f17370765cab\": container with ID starting with e8e0992b354db71a5769654d9ed76b1c5c5ea34eaf317624f8e5f17370765cab not found: ID does not exist" containerID="e8e0992b354db71a5769654d9ed76b1c5c5ea34eaf317624f8e5f17370765cab"
Jul 28 18:12:07.967027 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.966985    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8e0992b354db71a5769654d9ed76b1c5c5ea34eaf317624f8e5f17370765cab"} err="failed to get container status \"e8e0992b354db71a5769654d9ed76b1c5c5ea34eaf317624f8e5f17370765cab\": rpc error: code = NotFound desc = could not find container \"e8e0992b354db71a5769654d9ed76b1c5c5ea34eaf317624f8e5f17370765cab\": container with ID starting with e8e0992b354db71a5769654d9ed76b1c5c5ea34eaf317624f8e5f17370765cab not found: ID does not exist"
Jul 28 18:12:07.967027 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.966998    2581 scope.go:117] "RemoveContainer" containerID="952e05b522ad19383d982b59618e5435257e80a7a61ed277b6bfedcbabd15b3a"
Jul 28 18:12:07.967200 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:12:07.967181    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"952e05b522ad19383d982b59618e5435257e80a7a61ed277b6bfedcbabd15b3a\": container with ID starting with 952e05b522ad19383d982b59618e5435257e80a7a61ed277b6bfedcbabd15b3a not found: ID does not exist" containerID="952e05b522ad19383d982b59618e5435257e80a7a61ed277b6bfedcbabd15b3a"
Jul 28 18:12:07.967242 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.967205    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"952e05b522ad19383d982b59618e5435257e80a7a61ed277b6bfedcbabd15b3a"} err="failed to get container status \"952e05b522ad19383d982b59618e5435257e80a7a61ed277b6bfedcbabd15b3a\": rpc error: code = NotFound desc = could not find container \"952e05b522ad19383d982b59618e5435257e80a7a61ed277b6bfedcbabd15b3a\": container with ID starting with 952e05b522ad19383d982b59618e5435257e80a7a61ed277b6bfedcbabd15b3a not found: ID does not exist"
Jul 28 18:12:07.980419 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.980391    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"]
Jul 28 18:12:07.997314 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:07.997287    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-d9vxw"]
Jul 28 18:12:08.170968 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.170914    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:12:08.264067 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.264027    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:12:08.264377 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.264355    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"]
Jul 28 18:12:08.264608 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.264593    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e4e10cb-8be6-4573-bcb3-f3abbe356f79" containerName="kube-rbac-proxy"
Jul 28 18:12:08.264651 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.264610    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e4e10cb-8be6-4573-bcb3-f3abbe356f79" containerName="kube-rbac-proxy"
Jul 28 18:12:08.264651 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.264620    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d520360a-3ca3-4e45-8be8-074cc168bf57" containerName="kube-rbac-proxy"
Jul 28 18:12:08.264651 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.264625    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d520360a-3ca3-4e45-8be8-074cc168bf57" containerName="kube-rbac-proxy"
Jul 28 18:12:08.264651 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.264632    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e4e10cb-8be6-4573-bcb3-f3abbe356f79" containerName="kserve-container"
Jul 28 18:12:08.264651 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.264638    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e4e10cb-8be6-4573-bcb3-f3abbe356f79" containerName="kserve-container"
Jul 28 18:12:08.264651 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.264652    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d520360a-3ca3-4e45-8be8-074cc168bf57" containerName="kserve-container"
Jul 28 18:12:08.264845 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.264657    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d520360a-3ca3-4e45-8be8-074cc168bf57" containerName="kserve-container"
Jul 28 18:12:08.264845 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.264664    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d520360a-3ca3-4e45-8be8-074cc168bf57" containerName="storage-initializer"
Jul 28 18:12:08.264845 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.264669    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d520360a-3ca3-4e45-8be8-074cc168bf57" containerName="storage-initializer"
Jul 28 18:12:08.264845 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.264677    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e4e10cb-8be6-4573-bcb3-f3abbe356f79" containerName="storage-initializer"
Jul 28 18:12:08.264845 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.264682    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e4e10cb-8be6-4573-bcb3-f3abbe356f79" containerName="storage-initializer"
Jul 28 18:12:08.264845 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.264724    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d520360a-3ca3-4e45-8be8-074cc168bf57" containerName="kube-rbac-proxy"
Jul 28 18:12:08.264845 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.264732    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d520360a-3ca3-4e45-8be8-074cc168bf57" containerName="kserve-container"
Jul 28 18:12:08.264845 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.264739    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e4e10cb-8be6-4573-bcb3-f3abbe356f79" containerName="kserve-container"
Jul 28 18:12:08.264845 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.264743    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e4e10cb-8be6-4573-bcb3-f3abbe356f79" containerName="kube-rbac-proxy"
Jul 28 18:12:08.269076 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.269057    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"
Jul 28 18:12:08.272086 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.272060    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:12:08.272189 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.272060    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Jul 28 18:12:08.273112 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.273096    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:12:08.273320 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.273305    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:12:08.273414 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.273306    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:12:08.273481 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.273415    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 28 18:12:08.273558 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.273542    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 18:12:08.283605 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.283580    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"]
Jul 28 18:12:08.289839 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.289814    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e4e10cb-8be6-4573-bcb3-f3abbe356f79" path="/var/lib/kubelet/pods/5e4e10cb-8be6-4573-bcb3-f3abbe356f79/volumes"
Jul 28 18:12:08.365569 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.365521    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f38630f9-df8f-4227-896b-537d3b821a97-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59\" (UID: \"f38630f9-df8f-4227-896b-537d3b821a97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"
Jul 28 18:12:08.365569 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.365567    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f38630f9-df8f-4227-896b-537d3b821a97-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59\" (UID: \"f38630f9-df8f-4227-896b-537d3b821a97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"
Jul 28 18:12:08.365848 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.365641    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f38630f9-df8f-4227-896b-537d3b821a97-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59\" (UID: \"f38630f9-df8f-4227-896b-537d3b821a97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"
Jul 28 18:12:08.365848 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.365705    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lblw7\" (UniqueName: \"kubernetes.io/projected/f38630f9-df8f-4227-896b-537d3b821a97-kube-api-access-lblw7\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59\" (UID: \"f38630f9-df8f-4227-896b-537d3b821a97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"
Jul 28 18:12:08.365848 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.365743    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f38630f9-df8f-4227-896b-537d3b821a97-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59\" (UID: \"f38630f9-df8f-4227-896b-537d3b821a97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"
Jul 28 18:12:08.467074 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.466974    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f38630f9-df8f-4227-896b-537d3b821a97-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59\" (UID: \"f38630f9-df8f-4227-896b-537d3b821a97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"
Jul 28 18:12:08.467074 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.467022    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f38630f9-df8f-4227-896b-537d3b821a97-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59\" (UID: \"f38630f9-df8f-4227-896b-537d3b821a97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"
Jul 28 18:12:08.467074 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.467065    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f38630f9-df8f-4227-896b-537d3b821a97-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59\" (UID: \"f38630f9-df8f-4227-896b-537d3b821a97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"
Jul 28 18:12:08.467363 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.467094    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lblw7\" (UniqueName: \"kubernetes.io/projected/f38630f9-df8f-4227-896b-537d3b821a97-kube-api-access-lblw7\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59\" (UID: \"f38630f9-df8f-4227-896b-537d3b821a97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"
Jul 28 18:12:08.467363 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.467135    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f38630f9-df8f-4227-896b-537d3b821a97-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59\" (UID: \"f38630f9-df8f-4227-896b-537d3b821a97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"
Jul 28 18:12:08.467552 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.467517    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f38630f9-df8f-4227-896b-537d3b821a97-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59\" (UID: \"f38630f9-df8f-4227-896b-537d3b821a97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"
Jul 28 18:12:08.467696 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.467676    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f38630f9-df8f-4227-896b-537d3b821a97-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59\" (UID: \"f38630f9-df8f-4227-896b-537d3b821a97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"
Jul 28 18:12:08.467741 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.467720    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f38630f9-df8f-4227-896b-537d3b821a97-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59\" (UID: \"f38630f9-df8f-4227-896b-537d3b821a97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"
Jul 28 18:12:08.469593 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.469575    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f38630f9-df8f-4227-896b-537d3b821a97-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59\" (UID: \"f38630f9-df8f-4227-896b-537d3b821a97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"
Jul 28 18:12:08.477368 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.477346    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lblw7\" (UniqueName: \"kubernetes.io/projected/f38630f9-df8f-4227-896b-537d3b821a97-kube-api-access-lblw7\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59\" (UID: \"f38630f9-df8f-4227-896b-537d3b821a97\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"
Jul 28 18:12:08.579642 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.579607    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"
Jul 28 18:12:08.719252 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.719040    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"]
Jul 28 18:12:08.721669 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:12:08.721643    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf38630f9_df8f_4227_896b_537d3b821a97.slice/crio-076c43a3247732934568062129865ba4354a1f060e4eba5339d90a6f350ba229 WatchSource:0}: Error finding container 076c43a3247732934568062129865ba4354a1f060e4eba5339d90a6f350ba229: Status 404 returned error can't find the container with id 076c43a3247732934568062129865ba4354a1f060e4eba5339d90a6f350ba229
Jul 28 18:12:08.949476 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.949435    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59" event={"ID":"f38630f9-df8f-4227-896b-537d3b821a97","Type":"ContainerStarted","Data":"c66fc6e0397f1402ad2e02a24eacadafc7cdc5937b2245142202ba9adcb7b02a"}
Jul 28 18:12:08.949928 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:08.949507    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59" event={"ID":"f38630f9-df8f-4227-896b-537d3b821a97","Type":"ContainerStarted","Data":"076c43a3247732934568062129865ba4354a1f060e4eba5339d90a6f350ba229"}
Jul 28 18:12:12.968576 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:12.968543    2581 generic.go:356] "Generic (PLEG): container finished" podID="f38630f9-df8f-4227-896b-537d3b821a97" containerID="c66fc6e0397f1402ad2e02a24eacadafc7cdc5937b2245142202ba9adcb7b02a" exitCode=0
Jul 28 18:12:12.968995 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:12.968628    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59" event={"ID":"f38630f9-df8f-4227-896b-537d3b821a97","Type":"ContainerDied","Data":"c66fc6e0397f1402ad2e02a24eacadafc7cdc5937b2245142202ba9adcb7b02a"}
Jul 28 18:12:13.975021 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:13.974977    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59" event={"ID":"f38630f9-df8f-4227-896b-537d3b821a97","Type":"ContainerStarted","Data":"a02a38d14d3068adfd0bb79603e4aa150ae2a21b1784a5ae353571e2f964ace4"}
Jul 28 18:12:13.975459 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:13.975041    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59" event={"ID":"f38630f9-df8f-4227-896b-537d3b821a97","Type":"ContainerStarted","Data":"d69956bdd4f0de27149c5116904d8d318a538da77cc3a1a660892d2393b335b2"}
Jul 28 18:12:13.975459 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:13.975324    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"
Jul 28 18:12:13.975459 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:13.975385    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"
Jul 28 18:12:13.999830 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:13.999778    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59" podStartSLOduration=2.38797532 podStartE2EDuration="5.999763429s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.611788109s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:12:13.999113416 +0000 UTC m=+3782.273729850" watchObservedRunningTime="2026-07-28 18:12:13.999763429 +0000 UTC m=+3782.274379861"
Jul 28 18:12:19.984784 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:19.984754    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"
Jul 28 18:12:49.986666 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:49.986617    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59" podUID="f38630f9-df8f-4227-896b-537d3b821a97" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.53:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 28 18:12:59.986165 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:12:59.986119    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59" podUID="f38630f9-df8f-4227-896b-537d3b821a97" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.53:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 28 18:13:09.985452 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:09.985413    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59" podUID="f38630f9-df8f-4227-896b-537d3b821a97" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.53:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 28 18:13:19.985361 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:19.985318    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59" podUID="f38630f9-df8f-4227-896b-537d3b821a97" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.53:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 28 18:13:30.061780 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:30.061745    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"
Jul 28 18:13:38.296888 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:38.296805    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"]
Jul 28 18:13:38.297424 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:38.297122    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59" podUID="f38630f9-df8f-4227-896b-537d3b821a97" containerName="kserve-container" containerID="cri-o://d69956bdd4f0de27149c5116904d8d318a538da77cc3a1a660892d2393b335b2" gracePeriod=30
Jul 28 18:13:38.297424 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:38.297147    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59" podUID="f38630f9-df8f-4227-896b-537d3b821a97" containerName="kube-rbac-proxy" containerID="cri-o://a02a38d14d3068adfd0bb79603e4aa150ae2a21b1784a5ae353571e2f964ace4" gracePeriod=30
Jul 28 18:13:38.349863 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:38.349823    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:13:39.280494 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:39.280460    2581 generic.go:356] "Generic (PLEG): container finished" podID="f38630f9-df8f-4227-896b-537d3b821a97" containerID="a02a38d14d3068adfd0bb79603e4aa150ae2a21b1784a5ae353571e2f964ace4" exitCode=2
Jul 28 18:13:39.280670 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:39.280538    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59" event={"ID":"f38630f9-df8f-4227-896b-537d3b821a97","Type":"ContainerDied","Data":"a02a38d14d3068adfd0bb79603e4aa150ae2a21b1784a5ae353571e2f964ace4"}
Jul 28 18:13:39.979826 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:39.979783    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59" podUID="f38630f9-df8f-4227-896b-537d3b821a97" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Jul 28 18:13:39.986269 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:39.986239    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59" podUID="f38630f9-df8f-4227-896b-537d3b821a97" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.53:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 28 18:13:43.146572 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.146545    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"
Jul 28 18:13:43.175083 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.175048    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f38630f9-df8f-4227-896b-537d3b821a97-kserve-provision-location\") pod \"f38630f9-df8f-4227-896b-537d3b821a97\" (UID: \"f38630f9-df8f-4227-896b-537d3b821a97\") "
Jul 28 18:13:43.175268 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.175115    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f38630f9-df8f-4227-896b-537d3b821a97-proxy-tls\") pod \"f38630f9-df8f-4227-896b-537d3b821a97\" (UID: \"f38630f9-df8f-4227-896b-537d3b821a97\") "
Jul 28 18:13:43.175268 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.175143    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f38630f9-df8f-4227-896b-537d3b821a97-cabundle-cert\") pod \"f38630f9-df8f-4227-896b-537d3b821a97\" (UID: \"f38630f9-df8f-4227-896b-537d3b821a97\") "
Jul 28 18:13:43.175268 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.175208    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f38630f9-df8f-4227-896b-537d3b821a97-kube-rbac-proxy-sar-config\") pod \"f38630f9-df8f-4227-896b-537d3b821a97\" (UID: \"f38630f9-df8f-4227-896b-537d3b821a97\") "
Jul 28 18:13:43.175268 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.175233    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lblw7\" (UniqueName: \"kubernetes.io/projected/f38630f9-df8f-4227-896b-537d3b821a97-kube-api-access-lblw7\") pod \"f38630f9-df8f-4227-896b-537d3b821a97\" (UID: \"f38630f9-df8f-4227-896b-537d3b821a97\") "
Jul 28 18:13:43.175482 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.175393    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38630f9-df8f-4227-896b-537d3b821a97-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f38630f9-df8f-4227-896b-537d3b821a97" (UID: "f38630f9-df8f-4227-896b-537d3b821a97"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:13:43.175648 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.175621    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f38630f9-df8f-4227-896b-537d3b821a97-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f38630f9-df8f-4227-896b-537d3b821a97" (UID: "f38630f9-df8f-4227-896b-537d3b821a97"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:13:43.175648 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.175616    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f38630f9-df8f-4227-896b-537d3b821a97-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f38630f9-df8f-4227-896b-537d3b821a97" (UID: "f38630f9-df8f-4227-896b-537d3b821a97"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:13:43.177412 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.177374    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f38630f9-df8f-4227-896b-537d3b821a97-kube-api-access-lblw7" (OuterVolumeSpecName: "kube-api-access-lblw7") pod "f38630f9-df8f-4227-896b-537d3b821a97" (UID: "f38630f9-df8f-4227-896b-537d3b821a97"). InnerVolumeSpecName "kube-api-access-lblw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:13:43.177508 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.177448    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f38630f9-df8f-4227-896b-537d3b821a97-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f38630f9-df8f-4227-896b-537d3b821a97" (UID: "f38630f9-df8f-4227-896b-537d3b821a97"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:13:43.275911 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.275820    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f38630f9-df8f-4227-896b-537d3b821a97-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:13:43.275911 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.275852    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f38630f9-df8f-4227-896b-537d3b821a97-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:13:43.275911 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.275864    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f38630f9-df8f-4227-896b-537d3b821a97-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:13:43.275911 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.275873    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lblw7\" (UniqueName: \"kubernetes.io/projected/f38630f9-df8f-4227-896b-537d3b821a97-kube-api-access-lblw7\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:13:43.275911 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.275881    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f38630f9-df8f-4227-896b-537d3b821a97-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:13:43.296034 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.295996    2581 generic.go:356] "Generic (PLEG): container finished" podID="f38630f9-df8f-4227-896b-537d3b821a97" containerID="d69956bdd4f0de27149c5116904d8d318a538da77cc3a1a660892d2393b335b2" exitCode=0
Jul 28 18:13:43.296193 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.296059    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"
Jul 28 18:13:43.296193 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.296065    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59" event={"ID":"f38630f9-df8f-4227-896b-537d3b821a97","Type":"ContainerDied","Data":"d69956bdd4f0de27149c5116904d8d318a538da77cc3a1a660892d2393b335b2"}
Jul 28 18:13:43.296193 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.296127    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59" event={"ID":"f38630f9-df8f-4227-896b-537d3b821a97","Type":"ContainerDied","Data":"076c43a3247732934568062129865ba4354a1f060e4eba5339d90a6f350ba229"}
Jul 28 18:13:43.296193 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.296164    2581 scope.go:117] "RemoveContainer" containerID="a02a38d14d3068adfd0bb79603e4aa150ae2a21b1784a5ae353571e2f964ace4"
Jul 28 18:13:43.304852 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.304825    2581 scope.go:117] "RemoveContainer" containerID="d69956bdd4f0de27149c5116904d8d318a538da77cc3a1a660892d2393b335b2"
Jul 28 18:13:43.313329 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.313307    2581 scope.go:117] "RemoveContainer" containerID="c66fc6e0397f1402ad2e02a24eacadafc7cdc5937b2245142202ba9adcb7b02a"
Jul 28 18:13:43.322011 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.321985    2581 scope.go:117] "RemoveContainer" containerID="a02a38d14d3068adfd0bb79603e4aa150ae2a21b1784a5ae353571e2f964ace4"
Jul 28 18:13:43.322351 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:13:43.322326    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a02a38d14d3068adfd0bb79603e4aa150ae2a21b1784a5ae353571e2f964ace4\": container with ID starting with a02a38d14d3068adfd0bb79603e4aa150ae2a21b1784a5ae353571e2f964ace4 not found: ID does not exist" containerID="a02a38d14d3068adfd0bb79603e4aa150ae2a21b1784a5ae353571e2f964ace4"
Jul 28 18:13:43.322427 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.322364    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a02a38d14d3068adfd0bb79603e4aa150ae2a21b1784a5ae353571e2f964ace4"} err="failed to get container status \"a02a38d14d3068adfd0bb79603e4aa150ae2a21b1784a5ae353571e2f964ace4\": rpc error: code = NotFound desc = could not find container \"a02a38d14d3068adfd0bb79603e4aa150ae2a21b1784a5ae353571e2f964ace4\": container with ID starting with a02a38d14d3068adfd0bb79603e4aa150ae2a21b1784a5ae353571e2f964ace4 not found: ID does not exist"
Jul 28 18:13:43.322427 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.322391    2581 scope.go:117] "RemoveContainer" containerID="d69956bdd4f0de27149c5116904d8d318a538da77cc3a1a660892d2393b335b2"
Jul 28 18:13:43.322679 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:13:43.322659    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d69956bdd4f0de27149c5116904d8d318a538da77cc3a1a660892d2393b335b2\": container with ID starting with d69956bdd4f0de27149c5116904d8d318a538da77cc3a1a660892d2393b335b2 not found: ID does not exist" containerID="d69956bdd4f0de27149c5116904d8d318a538da77cc3a1a660892d2393b335b2"
Jul 28 18:13:43.322757 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.322684    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d69956bdd4f0de27149c5116904d8d318a538da77cc3a1a660892d2393b335b2"} err="failed to get container status \"d69956bdd4f0de27149c5116904d8d318a538da77cc3a1a660892d2393b335b2\": rpc error: code = NotFound desc = could not find container \"d69956bdd4f0de27149c5116904d8d318a538da77cc3a1a660892d2393b335b2\": container with ID starting with d69956bdd4f0de27149c5116904d8d318a538da77cc3a1a660892d2393b335b2 not found: ID does not exist"
Jul 28 18:13:43.322757 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.322701    2581 scope.go:117] "RemoveContainer" containerID="c66fc6e0397f1402ad2e02a24eacadafc7cdc5937b2245142202ba9adcb7b02a"
Jul 28 18:13:43.322990 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:13:43.322966    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c66fc6e0397f1402ad2e02a24eacadafc7cdc5937b2245142202ba9adcb7b02a\": container with ID starting with c66fc6e0397f1402ad2e02a24eacadafc7cdc5937b2245142202ba9adcb7b02a not found: ID does not exist" containerID="c66fc6e0397f1402ad2e02a24eacadafc7cdc5937b2245142202ba9adcb7b02a"
Jul 28 18:13:43.323047 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.322999    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c66fc6e0397f1402ad2e02a24eacadafc7cdc5937b2245142202ba9adcb7b02a"} err="failed to get container status \"c66fc6e0397f1402ad2e02a24eacadafc7cdc5937b2245142202ba9adcb7b02a\": rpc error: code = NotFound desc = could not find container \"c66fc6e0397f1402ad2e02a24eacadafc7cdc5937b2245142202ba9adcb7b02a\": container with ID starting with c66fc6e0397f1402ad2e02a24eacadafc7cdc5937b2245142202ba9adcb7b02a not found: ID does not exist"
Jul 28 18:13:43.325165 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.325141    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"]
Jul 28 18:13:43.333780 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.333755    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-gth59"]
Jul 28 18:13:43.382117 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.382082    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:13:43.504031 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.503994    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"]
Jul 28 18:13:43.504247 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.504235    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f38630f9-df8f-4227-896b-537d3b821a97" containerName="storage-initializer"
Jul 28 18:13:43.504292 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.504249    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f38630f9-df8f-4227-896b-537d3b821a97" containerName="storage-initializer"
Jul 28 18:13:43.504292 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.504264    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f38630f9-df8f-4227-896b-537d3b821a97" containerName="kube-rbac-proxy"
Jul 28 18:13:43.504292 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.504270    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f38630f9-df8f-4227-896b-537d3b821a97" containerName="kube-rbac-proxy"
Jul 28 18:13:43.504292 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.504281    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f38630f9-df8f-4227-896b-537d3b821a97" containerName="kserve-container"
Jul 28 18:13:43.504292 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.504287    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f38630f9-df8f-4227-896b-537d3b821a97" containerName="kserve-container"
Jul 28 18:13:43.504471 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.504330    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f38630f9-df8f-4227-896b-537d3b821a97" containerName="kserve-container"
Jul 28 18:13:43.504471 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.504338    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f38630f9-df8f-4227-896b-537d3b821a97" containerName="kube-rbac-proxy"
Jul 28 18:13:43.508632 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.508612    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"
Jul 28 18:13:43.511430 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.511404    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Jul 28 18:13:43.512152 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.512134    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 18:13:43.512499 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.512481    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:13:43.512592 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.512562    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:13:43.512689 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.512673    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:13:43.512965 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.512925    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:13:43.513012 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.512979    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Jul 28 18:13:43.533274 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.533186    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"]
Jul 28 18:13:43.577766 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.577729    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z\" (UID: \"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"
Jul 28 18:13:43.577766 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.577770    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z\" (UID: \"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"
Jul 28 18:13:43.578082 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.577858    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z\" (UID: \"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"
Jul 28 18:13:43.578082 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.577894    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phhnk\" (UniqueName: \"kubernetes.io/projected/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-kube-api-access-phhnk\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z\" (UID: \"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"
Jul 28 18:13:43.578082 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.577921    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z\" (UID: \"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"
Jul 28 18:13:43.598900 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.598870    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:13:43.678288 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.678249    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z\" (UID: \"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"
Jul 28 18:13:43.678288 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.678289    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phhnk\" (UniqueName: \"kubernetes.io/projected/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-kube-api-access-phhnk\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z\" (UID: \"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"
Jul 28 18:13:43.678539 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.678309    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z\" (UID: \"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"
Jul 28 18:13:43.678539 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.678356    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z\" (UID: \"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"
Jul 28 18:13:43.678539 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.678373    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z\" (UID: \"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"
Jul 28 18:13:43.678903 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.678876    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z\" (UID: \"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"
Jul 28 18:13:43.679267 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.679245    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z\" (UID: \"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"
Jul 28 18:13:43.679319 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.679246    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z\" (UID: \"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"
Jul 28 18:13:43.680702 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.680679    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z\" (UID: \"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"
Jul 28 18:13:43.688125 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.688099    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phhnk\" (UniqueName: \"kubernetes.io/projected/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-kube-api-access-phhnk\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z\" (UID: \"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"
Jul 28 18:13:43.819077 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.818969    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"
Jul 28 18:13:43.980003 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:43.979975    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"]
Jul 28 18:13:43.981608 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:13:43.981584    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8a5be9_b1bd_4ff7_9dce_f12dd52985c9.slice/crio-11b0b660ec53f02f16b0661067cd42ba448558626ab0502861e9aefd3b48156e WatchSource:0}: Error finding container 11b0b660ec53f02f16b0661067cd42ba448558626ab0502861e9aefd3b48156e: Status 404 returned error can't find the container with id 11b0b660ec53f02f16b0661067cd42ba448558626ab0502861e9aefd3b48156e
Jul 28 18:13:44.290480 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:44.290445    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f38630f9-df8f-4227-896b-537d3b821a97" path="/var/lib/kubelet/pods/f38630f9-df8f-4227-896b-537d3b821a97/volumes"
Jul 28 18:13:44.301562 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:44.301529    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z" event={"ID":"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9","Type":"ContainerStarted","Data":"524e410795674e5d257dc4101d21dd711b9bbfe657652295c65ae66debd53b53"}
Jul 28 18:13:44.301723 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:44.301584    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z" event={"ID":"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9","Type":"ContainerStarted","Data":"11b0b660ec53f02f16b0661067cd42ba448558626ab0502861e9aefd3b48156e"}
Jul 28 18:13:48.318121 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:48.318029    2581 generic.go:356] "Generic (PLEG): container finished" podID="8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9" containerID="524e410795674e5d257dc4101d21dd711b9bbfe657652295c65ae66debd53b53" exitCode=0
Jul 28 18:13:48.318121 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:48.318065    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z" event={"ID":"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9","Type":"ContainerDied","Data":"524e410795674e5d257dc4101d21dd711b9bbfe657652295c65ae66debd53b53"}
Jul 28 18:13:49.324115 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:49.324079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z" event={"ID":"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9","Type":"ContainerStarted","Data":"332d99d4aa7e36c53f0f57f3fb8246810e64eabda897d34a425dcef2a4edac62"}
Jul 28 18:13:49.324555 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:49.324150    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z" event={"ID":"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9","Type":"ContainerStarted","Data":"94cc50e995b425cff7723dc409dbb624349189c64a6c42c781754eb0c71d6bc4"}
Jul 28 18:13:49.324555 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:49.324361    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"
Jul 28 18:13:49.353458 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:49.353403    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z" podStartSLOduration=2.372465221 podStartE2EDuration="6.353385372s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.980920151s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:13:49.353316049 +0000 UTC m=+3877.627932481" watchObservedRunningTime="2026-07-28 18:13:49.353385372 +0000 UTC m=+3877.628001804"
Jul 28 18:13:50.327284 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:50.327255    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"
Jul 28 18:13:56.335708 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:13:56.335674    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"
Jul 28 18:14:08.910348 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:14:08.910309    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:14:12.419005 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:14:12.418973    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 18:14:12.423906 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:14:12.423884    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 18:14:26.337356 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:14:26.337315    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z" podUID="8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.54:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 18:14:36.336742 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:14:36.336696    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z" podUID="8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.54:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 18:14:46.336627 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:14:46.336585    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z" podUID="8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.54:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 18:14:50.998605 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:14:50.998564    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:14:56.337051 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:14:56.337003    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z" podUID="8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.54:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 18:15:06.362827 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:06.362726    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"
Jul 28 18:15:13.554265 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:13.554228    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"]
Jul 28 18:15:13.554720 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:13.554516    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z" podUID="8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9" containerName="kserve-container" containerID="cri-o://94cc50e995b425cff7723dc409dbb624349189c64a6c42c781754eb0c71d6bc4" gracePeriod=30
Jul 28 18:15:13.554720 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:13.554595    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z" podUID="8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9" containerName="kube-rbac-proxy" containerID="cri-o://332d99d4aa7e36c53f0f57f3fb8246810e64eabda897d34a425dcef2a4edac62" gracePeriod=30
Jul 28 18:15:13.579338 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:13.579308    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:15:14.620704 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:14.620671    2581 generic.go:356] "Generic (PLEG): container finished" podID="8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9" containerID="332d99d4aa7e36c53f0f57f3fb8246810e64eabda897d34a425dcef2a4edac62" exitCode=2
Jul 28 18:15:14.621124 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:14.620716    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z" event={"ID":"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9","Type":"ContainerDied","Data":"332d99d4aa7e36c53f0f57f3fb8246810e64eabda897d34a425dcef2a4edac62"}
Jul 28 18:15:16.330580 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:16.330534    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z" podUID="8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": dial tcp 10.133.0.54:8643: connect: connection refused"
Jul 28 18:15:16.337110 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:16.337074    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z" podUID="8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.54:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 28 18:15:18.637202 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:18.637106    2581 generic.go:356] "Generic (PLEG): container finished" podID="8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9" containerID="94cc50e995b425cff7723dc409dbb624349189c64a6c42c781754eb0c71d6bc4" exitCode=0
Jul 28 18:15:18.637202 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:18.637144    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z" event={"ID":"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9","Type":"ContainerDied","Data":"94cc50e995b425cff7723dc409dbb624349189c64a6c42c781754eb0c71d6bc4"}
Jul 28 18:15:18.652527 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:18.652489    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:15:18.701789 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:18.701763    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"
Jul 28 18:15:18.877529 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:18.877436    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-kserve-provision-location\") pod \"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9\" (UID: \"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9\") "
Jul 28 18:15:18.877529 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:18.877478    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-cabundle-cert\") pod \"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9\" (UID: \"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9\") "
Jul 28 18:15:18.877777 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:18.877542    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phhnk\" (UniqueName: \"kubernetes.io/projected/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-kube-api-access-phhnk\") pod \"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9\" (UID: \"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9\") "
Jul 28 18:15:18.877777 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:18.877570    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-proxy-tls\") pod \"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9\" (UID: \"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9\") "
Jul 28 18:15:18.877777 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:18.877587    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-kube-rbac-proxy-sar-config\") pod \"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9\" (UID: \"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9\") "
Jul 28 18:15:18.877974 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:18.877880    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9" (UID: "8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:15:18.878025 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:18.877984    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9" (UID: "8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:15:18.878025 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:18.877994    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9" (UID: "8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:15:18.879719 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:18.879696    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9" (UID: "8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:15:18.879778 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:18.879700    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-kube-api-access-phhnk" (OuterVolumeSpecName: "kube-api-access-phhnk") pod "8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9" (UID: "8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9"). InnerVolumeSpecName "kube-api-access-phhnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:15:18.957590 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:18.957558    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:15:18.978262 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:18.978223    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phhnk\" (UniqueName: \"kubernetes.io/projected/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-kube-api-access-phhnk\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:15:18.978262 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:18.978253    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:15:18.978262 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:18.978263    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:15:18.978262 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:18.978271    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:15:18.978531 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:18.978282    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:15:19.642981 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:19.642934    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"
Jul 28 18:15:19.643389 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:19.642937    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z" event={"ID":"8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9","Type":"ContainerDied","Data":"11b0b660ec53f02f16b0661067cd42ba448558626ab0502861e9aefd3b48156e"}
Jul 28 18:15:19.643389 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:19.643096    2581 scope.go:117] "RemoveContainer" containerID="332d99d4aa7e36c53f0f57f3fb8246810e64eabda897d34a425dcef2a4edac62"
Jul 28 18:15:19.651219 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:19.651199    2581 scope.go:117] "RemoveContainer" containerID="94cc50e995b425cff7723dc409dbb624349189c64a6c42c781754eb0c71d6bc4"
Jul 28 18:15:19.658473 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:19.658451    2581 scope.go:117] "RemoveContainer" containerID="524e410795674e5d257dc4101d21dd711b9bbfe657652295c65ae66debd53b53"
Jul 28 18:15:19.677788 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:19.677758    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"]
Jul 28 18:15:19.684153 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:19.684123    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b7b9z"]
Jul 28 18:15:20.289808 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:20.289773    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9" path="/var/lib/kubelet/pods/8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9/volumes"
Jul 28 18:15:20.844374 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:20.844337    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:15:25.873408 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:25.873362    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:15:25.907094 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:25.907061    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"]
Jul 28 18:15:25.907320 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:25.907307    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9" containerName="kserve-container"
Jul 28 18:15:25.907369 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:25.907322    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9" containerName="kserve-container"
Jul 28 18:15:25.907369 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:25.907333    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9" containerName="storage-initializer"
Jul 28 18:15:25.907369 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:25.907340    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9" containerName="storage-initializer"
Jul 28 18:15:25.907369 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:25.907354    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9" containerName="kube-rbac-proxy"
Jul 28 18:15:25.907369 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:25.907362    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9" containerName="kube-rbac-proxy"
Jul 28 18:15:25.907563 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:25.907399    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9" containerName="kserve-container"
Jul 28 18:15:25.907563 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:25.907409    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b8a5be9-b1bd-4ff7-9dce-f12dd52985c9" containerName="kube-rbac-proxy"
Jul 28 18:15:25.911862 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:25.911841    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"
Jul 28 18:15:25.914688 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:25.914660    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Jul 28 18:15:25.914822 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:25.914695    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:15:25.914822 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:25.914667    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:15:25.915178 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:25.915160    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:15:25.915275 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:25.915164    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Jul 28 18:15:25.915900 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:25.915880    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:15:25.915972 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:25.915894    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 18:15:25.923627 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:25.923604    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"]
Jul 28 18:15:26.018332 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:26.018294    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:15:26.026384 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:26.026356    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7aae5b8f-b8e1-4677-a321-d567979c8cd3-kserve-provision-location\") pod \"isvc-sklearn-predictor-6765cc767b-v2h4n\" (UID: \"7aae5b8f-b8e1-4677-a321-d567979c8cd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"
Jul 28 18:15:26.026484 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:26.026392    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7aae5b8f-b8e1-4677-a321-d567979c8cd3-cabundle-cert\") pod \"isvc-sklearn-predictor-6765cc767b-v2h4n\" (UID: \"7aae5b8f-b8e1-4677-a321-d567979c8cd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"
Jul 28 18:15:26.026484 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:26.026417    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzs9r\" (UniqueName: \"kubernetes.io/projected/7aae5b8f-b8e1-4677-a321-d567979c8cd3-kube-api-access-tzs9r\") pod \"isvc-sklearn-predictor-6765cc767b-v2h4n\" (UID: \"7aae5b8f-b8e1-4677-a321-d567979c8cd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"
Jul 28 18:15:26.026484 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:26.026475    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aae5b8f-b8e1-4677-a321-d567979c8cd3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-6765cc767b-v2h4n\" (UID: \"7aae5b8f-b8e1-4677-a321-d567979c8cd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"
Jul 28 18:15:26.026592 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:26.026498    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aae5b8f-b8e1-4677-a321-d567979c8cd3-proxy-tls\") pod \"isvc-sklearn-predictor-6765cc767b-v2h4n\" (UID: \"7aae5b8f-b8e1-4677-a321-d567979c8cd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"
Jul 28 18:15:26.126878 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:26.126793    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7aae5b8f-b8e1-4677-a321-d567979c8cd3-kserve-provision-location\") pod \"isvc-sklearn-predictor-6765cc767b-v2h4n\" (UID: \"7aae5b8f-b8e1-4677-a321-d567979c8cd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"
Jul 28 18:15:26.126878 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:26.126828    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7aae5b8f-b8e1-4677-a321-d567979c8cd3-cabundle-cert\") pod \"isvc-sklearn-predictor-6765cc767b-v2h4n\" (UID: \"7aae5b8f-b8e1-4677-a321-d567979c8cd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"
Jul 28 18:15:26.126878 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:26.126847    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzs9r\" (UniqueName: \"kubernetes.io/projected/7aae5b8f-b8e1-4677-a321-d567979c8cd3-kube-api-access-tzs9r\") pod \"isvc-sklearn-predictor-6765cc767b-v2h4n\" (UID: \"7aae5b8f-b8e1-4677-a321-d567979c8cd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"
Jul 28 18:15:26.127160 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:26.126881    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aae5b8f-b8e1-4677-a321-d567979c8cd3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-6765cc767b-v2h4n\" (UID: \"7aae5b8f-b8e1-4677-a321-d567979c8cd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"
Jul 28 18:15:26.127160 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:26.126907    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aae5b8f-b8e1-4677-a321-d567979c8cd3-proxy-tls\") pod \"isvc-sklearn-predictor-6765cc767b-v2h4n\" (UID: \"7aae5b8f-b8e1-4677-a321-d567979c8cd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"
Jul 28 18:15:26.127330 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:26.127308    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7aae5b8f-b8e1-4677-a321-d567979c8cd3-kserve-provision-location\") pod \"isvc-sklearn-predictor-6765cc767b-v2h4n\" (UID: \"7aae5b8f-b8e1-4677-a321-d567979c8cd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"
Jul 28 18:15:26.127570 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:26.127553    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aae5b8f-b8e1-4677-a321-d567979c8cd3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-6765cc767b-v2h4n\" (UID: \"7aae5b8f-b8e1-4677-a321-d567979c8cd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"
Jul 28 18:15:26.127668 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:26.127649    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7aae5b8f-b8e1-4677-a321-d567979c8cd3-cabundle-cert\") pod \"isvc-sklearn-predictor-6765cc767b-v2h4n\" (UID: \"7aae5b8f-b8e1-4677-a321-d567979c8cd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"
Jul 28 18:15:26.129507 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:26.129486    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aae5b8f-b8e1-4677-a321-d567979c8cd3-proxy-tls\") pod \"isvc-sklearn-predictor-6765cc767b-v2h4n\" (UID: \"7aae5b8f-b8e1-4677-a321-d567979c8cd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"
Jul 28 18:15:26.136772 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:26.136743    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzs9r\" (UniqueName: \"kubernetes.io/projected/7aae5b8f-b8e1-4677-a321-d567979c8cd3-kube-api-access-tzs9r\") pod \"isvc-sklearn-predictor-6765cc767b-v2h4n\" (UID: \"7aae5b8f-b8e1-4677-a321-d567979c8cd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"
Jul 28 18:15:26.222376 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:26.222321    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"
Jul 28 18:15:26.357033 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:26.357005    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"]
Jul 28 18:15:26.359509 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:15:26.359481    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aae5b8f_b8e1_4677_a321_d567979c8cd3.slice/crio-567c0fe4088dcb63df0c93f0db1fd1a8df9fba62c8e9890e4a9cbfc8248ded3d WatchSource:0}: Error finding container 567c0fe4088dcb63df0c93f0db1fd1a8df9fba62c8e9890e4a9cbfc8248ded3d: Status 404 returned error can't find the container with id 567c0fe4088dcb63df0c93f0db1fd1a8df9fba62c8e9890e4a9cbfc8248ded3d
Jul 28 18:15:26.671350 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:26.671308    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n" event={"ID":"7aae5b8f-b8e1-4677-a321-d567979c8cd3","Type":"ContainerStarted","Data":"1c19fea896a40c26f45943733d3c596a66204b417519a39b4d14c2c25bf3b310"}
Jul 28 18:15:26.671577 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:26.671373    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n" event={"ID":"7aae5b8f-b8e1-4677-a321-d567979c8cd3","Type":"ContainerStarted","Data":"567c0fe4088dcb63df0c93f0db1fd1a8df9fba62c8e9890e4a9cbfc8248ded3d"}
Jul 28 18:15:30.688186 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:30.688154    2581 generic.go:356] "Generic (PLEG): container finished" podID="7aae5b8f-b8e1-4677-a321-d567979c8cd3" containerID="1c19fea896a40c26f45943733d3c596a66204b417519a39b4d14c2c25bf3b310" exitCode=0
Jul 28 18:15:30.688710 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:30.688198    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n" event={"ID":"7aae5b8f-b8e1-4677-a321-d567979c8cd3","Type":"ContainerDied","Data":"1c19fea896a40c26f45943733d3c596a66204b417519a39b4d14c2c25bf3b310"}
Jul 28 18:15:31.693759 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:31.693723    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n" event={"ID":"7aae5b8f-b8e1-4677-a321-d567979c8cd3","Type":"ContainerStarted","Data":"d24f5fcab876d911389d3c8986b189b292727a82ad39e4052236c4b399bced39"}
Jul 28 18:15:31.694200 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:31.693810    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n" event={"ID":"7aae5b8f-b8e1-4677-a321-d567979c8cd3","Type":"ContainerStarted","Data":"69124332268d18a73de34bd1161c18e05560bc69704049edc1ec6b2e29235a0f"}
Jul 28 18:15:31.694200 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:31.694100    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"
Jul 28 18:15:31.694287 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:31.694239    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"
Jul 28 18:15:31.695611 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:31.695585    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n" podUID="7aae5b8f-b8e1-4677-a321-d567979c8cd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 18:15:31.717547 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:31.717499    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n" podStartSLOduration=2.777676438 podStartE2EDuration="6.717485833s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.939809395s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:15:31.716331039 +0000 UTC m=+3979.990947497" watchObservedRunningTime="2026-07-28 18:15:31.717485833 +0000 UTC m=+3979.992102267"
Jul 28 18:15:32.697957 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:32.697902    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n" podUID="7aae5b8f-b8e1-4677-a321-d567979c8cd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 18:15:37.702124 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:37.702090    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"
Jul 28 18:15:37.702657 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:37.702632    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n" podUID="7aae5b8f-b8e1-4677-a321-d567979c8cd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 18:15:47.702821 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:47.702776    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n" podUID="7aae5b8f-b8e1-4677-a321-d567979c8cd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 18:15:57.703497 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:15:57.703457    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n" podUID="7aae5b8f-b8e1-4677-a321-d567979c8cd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 18:16:07.702918 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:16:07.702877    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n" podUID="7aae5b8f-b8e1-4677-a321-d567979c8cd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 18:16:17.702985 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:16:17.702925    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n" podUID="7aae5b8f-b8e1-4677-a321-d567979c8cd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 18:16:27.703423 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:16:27.703377    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n" podUID="7aae5b8f-b8e1-4677-a321-d567979c8cd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 18:16:37.702804 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:16:37.702714    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n" podUID="7aae5b8f-b8e1-4677-a321-d567979c8cd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 18:16:47.703984 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:16:47.703927    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"
Jul 28 18:16:56.052234 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:16:56.052191    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"]
Jul 28 18:16:56.052803 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:16:56.052444    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n" podUID="7aae5b8f-b8e1-4677-a321-d567979c8cd3" containerName="kserve-container" containerID="cri-o://69124332268d18a73de34bd1161c18e05560bc69704049edc1ec6b2e29235a0f" gracePeriod=30
Jul 28 18:16:56.052803 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:16:56.052497    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n" podUID="7aae5b8f-b8e1-4677-a321-d567979c8cd3" containerName="kube-rbac-proxy" containerID="cri-o://d24f5fcab876d911389d3c8986b189b292727a82ad39e4052236c4b399bced39" gracePeriod=30
Jul 28 18:16:56.093916 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:16:56.093875    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:16:56.999576 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:16:56.999544    2581 generic.go:356] "Generic (PLEG): container finished" podID="7aae5b8f-b8e1-4677-a321-d567979c8cd3" containerID="d24f5fcab876d911389d3c8986b189b292727a82ad39e4052236c4b399bced39" exitCode=2
Jul 28 18:16:56.999576 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:16:56.999581    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n" event={"ID":"7aae5b8f-b8e1-4677-a321-d567979c8cd3","Type":"ContainerDied","Data":"d24f5fcab876d911389d3c8986b189b292727a82ad39e4052236c4b399bced39"}
Jul 28 18:16:57.698814 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:16:57.698761    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n" podUID="7aae5b8f-b8e1-4677-a321-d567979c8cd3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.55:8643/healthz\": dial tcp 10.133.0.55:8643: connect: connection refused"
Jul 28 18:16:57.702651 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:16:57.702615    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n" podUID="7aae5b8f-b8e1-4677-a321-d567979c8cd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 28 18:17:00.891102 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:00.891066    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"
Jul 28 18:17:00.936567 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:00.936472    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aae5b8f-b8e1-4677-a321-d567979c8cd3-kube-rbac-proxy-sar-config\") pod \"7aae5b8f-b8e1-4677-a321-d567979c8cd3\" (UID: \"7aae5b8f-b8e1-4677-a321-d567979c8cd3\") "
Jul 28 18:17:00.936567 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:00.936554    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7aae5b8f-b8e1-4677-a321-d567979c8cd3-kserve-provision-location\") pod \"7aae5b8f-b8e1-4677-a321-d567979c8cd3\" (UID: \"7aae5b8f-b8e1-4677-a321-d567979c8cd3\") "
Jul 28 18:17:00.936807 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:00.936660    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aae5b8f-b8e1-4677-a321-d567979c8cd3-proxy-tls\") pod \"7aae5b8f-b8e1-4677-a321-d567979c8cd3\" (UID: \"7aae5b8f-b8e1-4677-a321-d567979c8cd3\") "
Jul 28 18:17:00.936807 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:00.936717    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7aae5b8f-b8e1-4677-a321-d567979c8cd3-cabundle-cert\") pod \"7aae5b8f-b8e1-4677-a321-d567979c8cd3\" (UID: \"7aae5b8f-b8e1-4677-a321-d567979c8cd3\") "
Jul 28 18:17:00.936973 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:00.936925    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aae5b8f-b8e1-4677-a321-d567979c8cd3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7aae5b8f-b8e1-4677-a321-d567979c8cd3" (UID: "7aae5b8f-b8e1-4677-a321-d567979c8cd3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:17:00.937035 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:00.936988    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aae5b8f-b8e1-4677-a321-d567979c8cd3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7aae5b8f-b8e1-4677-a321-d567979c8cd3" (UID: "7aae5b8f-b8e1-4677-a321-d567979c8cd3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:17:00.937035 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:00.936999    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aae5b8f-b8e1-4677-a321-d567979c8cd3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7aae5b8f-b8e1-4677-a321-d567979c8cd3" (UID: "7aae5b8f-b8e1-4677-a321-d567979c8cd3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:17:00.938648 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:00.938624    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aae5b8f-b8e1-4677-a321-d567979c8cd3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7aae5b8f-b8e1-4677-a321-d567979c8cd3" (UID: "7aae5b8f-b8e1-4677-a321-d567979c8cd3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:17:01.014385 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.014347    2581 generic.go:356] "Generic (PLEG): container finished" podID="7aae5b8f-b8e1-4677-a321-d567979c8cd3" containerID="69124332268d18a73de34bd1161c18e05560bc69704049edc1ec6b2e29235a0f" exitCode=0
Jul 28 18:17:01.014544 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.014412    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"
Jul 28 18:17:01.014544 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.014425    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n" event={"ID":"7aae5b8f-b8e1-4677-a321-d567979c8cd3","Type":"ContainerDied","Data":"69124332268d18a73de34bd1161c18e05560bc69704049edc1ec6b2e29235a0f"}
Jul 28 18:17:01.014544 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.014490    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n" event={"ID":"7aae5b8f-b8e1-4677-a321-d567979c8cd3","Type":"ContainerDied","Data":"567c0fe4088dcb63df0c93f0db1fd1a8df9fba62c8e9890e4a9cbfc8248ded3d"}
Jul 28 18:17:01.014544 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.014519    2581 scope.go:117] "RemoveContainer" containerID="d24f5fcab876d911389d3c8986b189b292727a82ad39e4052236c4b399bced39"
Jul 28 18:17:01.022263 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.022246    2581 scope.go:117] "RemoveContainer" containerID="69124332268d18a73de34bd1161c18e05560bc69704049edc1ec6b2e29235a0f"
Jul 28 18:17:01.029286 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.029268    2581 scope.go:117] "RemoveContainer" containerID="1c19fea896a40c26f45943733d3c596a66204b417519a39b4d14c2c25bf3b310"
Jul 28 18:17:01.036590 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.036572    2581 scope.go:117] "RemoveContainer" containerID="d24f5fcab876d911389d3c8986b189b292727a82ad39e4052236c4b399bced39"
Jul 28 18:17:01.036892 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:17:01.036870    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d24f5fcab876d911389d3c8986b189b292727a82ad39e4052236c4b399bced39\": container with ID starting with d24f5fcab876d911389d3c8986b189b292727a82ad39e4052236c4b399bced39 not found: ID does not exist" containerID="d24f5fcab876d911389d3c8986b189b292727a82ad39e4052236c4b399bced39"
Jul 28 18:17:01.037015 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.036907    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d24f5fcab876d911389d3c8986b189b292727a82ad39e4052236c4b399bced39"} err="failed to get container status \"d24f5fcab876d911389d3c8986b189b292727a82ad39e4052236c4b399bced39\": rpc error: code = NotFound desc = could not find container \"d24f5fcab876d911389d3c8986b189b292727a82ad39e4052236c4b399bced39\": container with ID starting with d24f5fcab876d911389d3c8986b189b292727a82ad39e4052236c4b399bced39 not found: ID does not exist"
Jul 28 18:17:01.037015 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.036928    2581 scope.go:117] "RemoveContainer" containerID="69124332268d18a73de34bd1161c18e05560bc69704049edc1ec6b2e29235a0f"
Jul 28 18:17:01.037186 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.037170    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzs9r\" (UniqueName: \"kubernetes.io/projected/7aae5b8f-b8e1-4677-a321-d567979c8cd3-kube-api-access-tzs9r\") pod \"7aae5b8f-b8e1-4677-a321-d567979c8cd3\" (UID: \"7aae5b8f-b8e1-4677-a321-d567979c8cd3\") "
Jul 28 18:17:01.037253 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:17:01.037233    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69124332268d18a73de34bd1161c18e05560bc69704049edc1ec6b2e29235a0f\": container with ID starting with 69124332268d18a73de34bd1161c18e05560bc69704049edc1ec6b2e29235a0f not found: ID does not exist" containerID="69124332268d18a73de34bd1161c18e05560bc69704049edc1ec6b2e29235a0f"
Jul 28 18:17:01.037295 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.037265    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69124332268d18a73de34bd1161c18e05560bc69704049edc1ec6b2e29235a0f"} err="failed to get container status \"69124332268d18a73de34bd1161c18e05560bc69704049edc1ec6b2e29235a0f\": rpc error: code = NotFound desc = could not find container \"69124332268d18a73de34bd1161c18e05560bc69704049edc1ec6b2e29235a0f\": container with ID starting with 69124332268d18a73de34bd1161c18e05560bc69704049edc1ec6b2e29235a0f not found: ID does not exist"
Jul 28 18:17:01.037295 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.037289    2581 scope.go:117] "RemoveContainer" containerID="1c19fea896a40c26f45943733d3c596a66204b417519a39b4d14c2c25bf3b310"
Jul 28 18:17:01.037366 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.037309    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7aae5b8f-b8e1-4677-a321-d567979c8cd3-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:17:01.037366 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.037322    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aae5b8f-b8e1-4677-a321-d567979c8cd3-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:17:01.037366 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.037334    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7aae5b8f-b8e1-4677-a321-d567979c8cd3-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:17:01.037366 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.037346    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aae5b8f-b8e1-4677-a321-d567979c8cd3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:17:01.037619 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:17:01.037594    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c19fea896a40c26f45943733d3c596a66204b417519a39b4d14c2c25bf3b310\": container with ID starting with 1c19fea896a40c26f45943733d3c596a66204b417519a39b4d14c2c25bf3b310 not found: ID does not exist" containerID="1c19fea896a40c26f45943733d3c596a66204b417519a39b4d14c2c25bf3b310"
Jul 28 18:17:01.037666 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.037626    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c19fea896a40c26f45943733d3c596a66204b417519a39b4d14c2c25bf3b310"} err="failed to get container status \"1c19fea896a40c26f45943733d3c596a66204b417519a39b4d14c2c25bf3b310\": rpc error: code = NotFound desc = could not find container \"1c19fea896a40c26f45943733d3c596a66204b417519a39b4d14c2c25bf3b310\": container with ID starting with 1c19fea896a40c26f45943733d3c596a66204b417519a39b4d14c2c25bf3b310 not found: ID does not exist"
Jul 28 18:17:01.039190 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.039165    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aae5b8f-b8e1-4677-a321-d567979c8cd3-kube-api-access-tzs9r" (OuterVolumeSpecName: "kube-api-access-tzs9r") pod "7aae5b8f-b8e1-4677-a321-d567979c8cd3" (UID: "7aae5b8f-b8e1-4677-a321-d567979c8cd3"). InnerVolumeSpecName "kube-api-access-tzs9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:17:01.138309 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.138274    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzs9r\" (UniqueName: \"kubernetes.io/projected/7aae5b8f-b8e1-4677-a321-d567979c8cd3-kube-api-access-tzs9r\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:17:01.221148 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.221055    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:17:01.333201 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.333168    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:17:01.363709 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.363665    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"]
Jul 28 18:17:01.388448 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.388408    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-v2h4n"]
Jul 28 18:17:01.398519 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.398486    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"]
Jul 28 18:17:01.398732 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.398720    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aae5b8f-b8e1-4677-a321-d567979c8cd3" containerName="kserve-container"
Jul 28 18:17:01.398776 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.398734    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aae5b8f-b8e1-4677-a321-d567979c8cd3" containerName="kserve-container"
Jul 28 18:17:01.398776 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.398743    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aae5b8f-b8e1-4677-a321-d567979c8cd3" containerName="storage-initializer"
Jul 28 18:17:01.398776 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.398749    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aae5b8f-b8e1-4677-a321-d567979c8cd3" containerName="storage-initializer"
Jul 28 18:17:01.398776 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.398759    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aae5b8f-b8e1-4677-a321-d567979c8cd3" containerName="kube-rbac-proxy"
Jul 28 18:17:01.398776 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.398765    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aae5b8f-b8e1-4677-a321-d567979c8cd3" containerName="kube-rbac-proxy"
Jul 28 18:17:01.398935 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.398810    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aae5b8f-b8e1-4677-a321-d567979c8cd3" containerName="kube-rbac-proxy"
Jul 28 18:17:01.398935 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.398819    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aae5b8f-b8e1-4677-a321-d567979c8cd3" containerName="kserve-container"
Jul 28 18:17:01.403269 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.403248    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"
Jul 28 18:17:01.406614 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.406584    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 28 18:17:01.406725 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.406663    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Jul 28 18:17:01.407106 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.407087    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:17:01.407275 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.407253    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 18:17:01.407375 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.407350    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:17:01.407501 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.407485    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:17:01.407606 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.407590    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:17:01.423595 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.423562    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"]
Jul 28 18:17:01.440555 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.440514    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l2vb\" (UniqueName: \"kubernetes.io/projected/46e76097-4dbc-426e-bac4-7df016937d30-kube-api-access-5l2vb\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2\" (UID: \"46e76097-4dbc-426e-bac4-7df016937d30\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"
Jul 28 18:17:01.440555 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.440556    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46e76097-4dbc-426e-bac4-7df016937d30-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2\" (UID: \"46e76097-4dbc-426e-bac4-7df016937d30\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"
Jul 28 18:17:01.440782 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.440580    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46e76097-4dbc-426e-bac4-7df016937d30-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2\" (UID: \"46e76097-4dbc-426e-bac4-7df016937d30\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"
Jul 28 18:17:01.440782 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.440602    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46e76097-4dbc-426e-bac4-7df016937d30-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2\" (UID: \"46e76097-4dbc-426e-bac4-7df016937d30\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"
Jul 28 18:17:01.440782 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.440627    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46e76097-4dbc-426e-bac4-7df016937d30-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2\" (UID: \"46e76097-4dbc-426e-bac4-7df016937d30\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"
Jul 28 18:17:01.541687 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.541578    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l2vb\" (UniqueName: \"kubernetes.io/projected/46e76097-4dbc-426e-bac4-7df016937d30-kube-api-access-5l2vb\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2\" (UID: \"46e76097-4dbc-426e-bac4-7df016937d30\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"
Jul 28 18:17:01.541687 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.541626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46e76097-4dbc-426e-bac4-7df016937d30-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2\" (UID: \"46e76097-4dbc-426e-bac4-7df016937d30\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"
Jul 28 18:17:01.541687 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.541650    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46e76097-4dbc-426e-bac4-7df016937d30-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2\" (UID: \"46e76097-4dbc-426e-bac4-7df016937d30\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"
Jul 28 18:17:01.542039 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.541716    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46e76097-4dbc-426e-bac4-7df016937d30-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2\" (UID: \"46e76097-4dbc-426e-bac4-7df016937d30\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"
Jul 28 18:17:01.542039 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.541781    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46e76097-4dbc-426e-bac4-7df016937d30-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2\" (UID: \"46e76097-4dbc-426e-bac4-7df016937d30\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"
Jul 28 18:17:01.542250 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.542226    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46e76097-4dbc-426e-bac4-7df016937d30-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2\" (UID: \"46e76097-4dbc-426e-bac4-7df016937d30\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"
Jul 28 18:17:01.542409 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.542391    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46e76097-4dbc-426e-bac4-7df016937d30-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2\" (UID: \"46e76097-4dbc-426e-bac4-7df016937d30\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"
Jul 28 18:17:01.542477 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.542459    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46e76097-4dbc-426e-bac4-7df016937d30-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2\" (UID: \"46e76097-4dbc-426e-bac4-7df016937d30\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"
Jul 28 18:17:01.544194 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.544175    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46e76097-4dbc-426e-bac4-7df016937d30-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2\" (UID: \"46e76097-4dbc-426e-bac4-7df016937d30\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"
Jul 28 18:17:01.551043 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.551023    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l2vb\" (UniqueName: \"kubernetes.io/projected/46e76097-4dbc-426e-bac4-7df016937d30-kube-api-access-5l2vb\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2\" (UID: \"46e76097-4dbc-426e-bac4-7df016937d30\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"
Jul 28 18:17:01.713701 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.713655    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"
Jul 28 18:17:01.853302 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:01.853275    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"]
Jul 28 18:17:01.855337 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:17:01.855306    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46e76097_4dbc_426e_bac4_7df016937d30.slice/crio-7b76fe9ef4aef6dc593be59cae3ec4cbc3d0a708567b415c3d657cd469ca78a6 WatchSource:0}: Error finding container 7b76fe9ef4aef6dc593be59cae3ec4cbc3d0a708567b415c3d657cd469ca78a6: Status 404 returned error can't find the container with id 7b76fe9ef4aef6dc593be59cae3ec4cbc3d0a708567b415c3d657cd469ca78a6
Jul 28 18:17:02.020803 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:02.020768    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2" event={"ID":"46e76097-4dbc-426e-bac4-7df016937d30","Type":"ContainerStarted","Data":"ce69d9551ecf724f2a41029ecfe19ab7521de155854b99b94f45527ef8aa9ebf"}
Jul 28 18:17:02.021212 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:02.020836    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2" event={"ID":"46e76097-4dbc-426e-bac4-7df016937d30","Type":"ContainerStarted","Data":"7b76fe9ef4aef6dc593be59cae3ec4cbc3d0a708567b415c3d657cd469ca78a6"}
Jul 28 18:17:02.291730 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:02.291688    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7aae5b8f-b8e1-4677-a321-d567979c8cd3" path="/var/lib/kubelet/pods/7aae5b8f-b8e1-4677-a321-d567979c8cd3/volumes"
Jul 28 18:17:06.036452 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:06.036418    2581 generic.go:356] "Generic (PLEG): container finished" podID="46e76097-4dbc-426e-bac4-7df016937d30" containerID="ce69d9551ecf724f2a41029ecfe19ab7521de155854b99b94f45527ef8aa9ebf" exitCode=0
Jul 28 18:17:06.036452 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:06.036456    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2" event={"ID":"46e76097-4dbc-426e-bac4-7df016937d30","Type":"ContainerDied","Data":"ce69d9551ecf724f2a41029ecfe19ab7521de155854b99b94f45527ef8aa9ebf"}
Jul 28 18:17:07.041559 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:07.041524    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2" event={"ID":"46e76097-4dbc-426e-bac4-7df016937d30","Type":"ContainerStarted","Data":"3a1d460ecb45a9ab6c2387f981bc9e1efd8a13fa0824a9297ae28c8eb046cf34"}
Jul 28 18:17:07.041986 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:07.041590    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2" event={"ID":"46e76097-4dbc-426e-bac4-7df016937d30","Type":"ContainerStarted","Data":"e2bb94ca0f8afd062a203f27205716f1518ce2027bf01357ae553842b6c0f69e"}
Jul 28 18:17:07.042742 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:07.042712    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"
Jul 28 18:17:07.042865 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:07.042837    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"
Jul 28 18:17:07.066777 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:07.066715    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2" podStartSLOduration=2.069493205 podStartE2EDuration="6.066700202s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.997206997s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:17:07.065410416 +0000 UTC m=+4075.340026882" watchObservedRunningTime="2026-07-28 18:17:07.066700202 +0000 UTC m=+4075.341316633"
Jul 28 18:17:13.050540 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:13.050455    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"
Jul 28 18:17:43.058278 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:43.058233    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2" podUID="46e76097-4dbc-426e-bac4-7df016937d30" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 28 18:17:53.053004 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:17:53.052965    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"
Jul 28 18:18:01.287359 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:01.287316    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"]
Jul 28 18:18:01.287803 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:01.287627    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2" podUID="46e76097-4dbc-426e-bac4-7df016937d30" containerName="kserve-container" containerID="cri-o://e2bb94ca0f8afd062a203f27205716f1518ce2027bf01357ae553842b6c0f69e" gracePeriod=30
Jul 28 18:18:01.287803 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:01.287649    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2" podUID="46e76097-4dbc-426e-bac4-7df016937d30" containerName="kube-rbac-proxy" containerID="cri-o://3a1d460ecb45a9ab6c2387f981bc9e1efd8a13fa0824a9297ae28c8eb046cf34" gracePeriod=30
Jul 28 18:18:01.319925 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:01.319892    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:18:02.233627 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:02.233588    2581 generic.go:356] "Generic (PLEG): container finished" podID="46e76097-4dbc-426e-bac4-7df016937d30" containerID="3a1d460ecb45a9ab6c2387f981bc9e1efd8a13fa0824a9297ae28c8eb046cf34" exitCode=2
Jul 28 18:18:02.233809 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:02.233645    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2" event={"ID":"46e76097-4dbc-426e-bac4-7df016937d30","Type":"ContainerDied","Data":"3a1d460ecb45a9ab6c2387f981bc9e1efd8a13fa0824a9297ae28c8eb046cf34"}
Jul 28 18:18:03.046531 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:03.046486    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2" podUID="46e76097-4dbc-426e-bac4-7df016937d30" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.56:8643/healthz\": dial tcp 10.133.0.56:8643: connect: connection refused"
Jul 28 18:18:04.092143 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:04.092095    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2" podUID="46e76097-4dbc-426e-bac4-7df016937d30" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.56:8080/v2/models/sklearn-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 18:18:06.385353 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:06.385257    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:18:06.591924 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:06.591885    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:18:08.045812 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:08.045770    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2" podUID="46e76097-4dbc-426e-bac4-7df016937d30" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.56:8643/healthz\": dial tcp 10.133.0.56:8643: connect: connection refused"
Jul 28 18:18:08.836998 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:08.836973    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"
Jul 28 18:18:08.967829 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:08.967738    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46e76097-4dbc-426e-bac4-7df016937d30-cabundle-cert\") pod \"46e76097-4dbc-426e-bac4-7df016937d30\" (UID: \"46e76097-4dbc-426e-bac4-7df016937d30\") "
Jul 28 18:18:08.968023 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:08.967833    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46e76097-4dbc-426e-bac4-7df016937d30-kserve-provision-location\") pod \"46e76097-4dbc-426e-bac4-7df016937d30\" (UID: \"46e76097-4dbc-426e-bac4-7df016937d30\") "
Jul 28 18:18:08.968023 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:08.967859    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5l2vb\" (UniqueName: \"kubernetes.io/projected/46e76097-4dbc-426e-bac4-7df016937d30-kube-api-access-5l2vb\") pod \"46e76097-4dbc-426e-bac4-7df016937d30\" (UID: \"46e76097-4dbc-426e-bac4-7df016937d30\") "
Jul 28 18:18:08.968023 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:08.967897    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46e76097-4dbc-426e-bac4-7df016937d30-kube-rbac-proxy-sar-config\") pod \"46e76097-4dbc-426e-bac4-7df016937d30\" (UID: \"46e76097-4dbc-426e-bac4-7df016937d30\") "
Jul 28 18:18:08.968023 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:08.967930    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46e76097-4dbc-426e-bac4-7df016937d30-proxy-tls\") pod \"46e76097-4dbc-426e-bac4-7df016937d30\" (UID: \"46e76097-4dbc-426e-bac4-7df016937d30\") "
Jul 28 18:18:08.968288 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:08.968258    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46e76097-4dbc-426e-bac4-7df016937d30-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "46e76097-4dbc-426e-bac4-7df016937d30" (UID: "46e76097-4dbc-426e-bac4-7df016937d30"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:18:08.968332 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:08.968276    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e76097-4dbc-426e-bac4-7df016937d30-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "46e76097-4dbc-426e-bac4-7df016937d30" (UID: "46e76097-4dbc-426e-bac4-7df016937d30"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:18:08.968332 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:08.968302    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46e76097-4dbc-426e-bac4-7df016937d30-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "46e76097-4dbc-426e-bac4-7df016937d30" (UID: "46e76097-4dbc-426e-bac4-7df016937d30"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:18:08.970055 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:08.970026    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46e76097-4dbc-426e-bac4-7df016937d30-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "46e76097-4dbc-426e-bac4-7df016937d30" (UID: "46e76097-4dbc-426e-bac4-7df016937d30"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:18:08.970172 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:08.970129    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e76097-4dbc-426e-bac4-7df016937d30-kube-api-access-5l2vb" (OuterVolumeSpecName: "kube-api-access-5l2vb") pod "46e76097-4dbc-426e-bac4-7df016937d30" (UID: "46e76097-4dbc-426e-bac4-7df016937d30"). InnerVolumeSpecName "kube-api-access-5l2vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:18:09.069280 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:09.069232    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/46e76097-4dbc-426e-bac4-7df016937d30-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:18:09.069280 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:09.069269    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/46e76097-4dbc-426e-bac4-7df016937d30-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:18:09.069280 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:09.069279    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/46e76097-4dbc-426e-bac4-7df016937d30-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:18:09.069280 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:09.069287    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/46e76097-4dbc-426e-bac4-7df016937d30-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:18:09.069280 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:09.069296    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l2vb\" (UniqueName: \"kubernetes.io/projected/46e76097-4dbc-426e-bac4-7df016937d30-kube-api-access-5l2vb\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:18:09.260148 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:09.260050    2581 generic.go:356] "Generic (PLEG): container finished" podID="46e76097-4dbc-426e-bac4-7df016937d30" containerID="e2bb94ca0f8afd062a203f27205716f1518ce2027bf01357ae553842b6c0f69e" exitCode=0
Jul 28 18:18:09.260148 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:09.260129    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"
Jul 28 18:18:09.260386 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:09.260140    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2" event={"ID":"46e76097-4dbc-426e-bac4-7df016937d30","Type":"ContainerDied","Data":"e2bb94ca0f8afd062a203f27205716f1518ce2027bf01357ae553842b6c0f69e"}
Jul 28 18:18:09.260386 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:09.260247    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2" event={"ID":"46e76097-4dbc-426e-bac4-7df016937d30","Type":"ContainerDied","Data":"7b76fe9ef4aef6dc593be59cae3ec4cbc3d0a708567b415c3d657cd469ca78a6"}
Jul 28 18:18:09.260386 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:09.260287    2581 scope.go:117] "RemoveContainer" containerID="3a1d460ecb45a9ab6c2387f981bc9e1efd8a13fa0824a9297ae28c8eb046cf34"
Jul 28 18:18:09.268971 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:09.268932    2581 scope.go:117] "RemoveContainer" containerID="e2bb94ca0f8afd062a203f27205716f1518ce2027bf01357ae553842b6c0f69e"
Jul 28 18:18:09.276167 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:09.276150    2581 scope.go:117] "RemoveContainer" containerID="ce69d9551ecf724f2a41029ecfe19ab7521de155854b99b94f45527ef8aa9ebf"
Jul 28 18:18:09.283322 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:09.283303    2581 scope.go:117] "RemoveContainer" containerID="3a1d460ecb45a9ab6c2387f981bc9e1efd8a13fa0824a9297ae28c8eb046cf34"
Jul 28 18:18:09.283575 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:18:09.283555    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a1d460ecb45a9ab6c2387f981bc9e1efd8a13fa0824a9297ae28c8eb046cf34\": container with ID starting with 3a1d460ecb45a9ab6c2387f981bc9e1efd8a13fa0824a9297ae28c8eb046cf34 not found: ID does not exist" containerID="3a1d460ecb45a9ab6c2387f981bc9e1efd8a13fa0824a9297ae28c8eb046cf34"
Jul 28 18:18:09.283615 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:09.283588    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a1d460ecb45a9ab6c2387f981bc9e1efd8a13fa0824a9297ae28c8eb046cf34"} err="failed to get container status \"3a1d460ecb45a9ab6c2387f981bc9e1efd8a13fa0824a9297ae28c8eb046cf34\": rpc error: code = NotFound desc = could not find container \"3a1d460ecb45a9ab6c2387f981bc9e1efd8a13fa0824a9297ae28c8eb046cf34\": container with ID starting with 3a1d460ecb45a9ab6c2387f981bc9e1efd8a13fa0824a9297ae28c8eb046cf34 not found: ID does not exist"
Jul 28 18:18:09.283656 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:09.283615    2581 scope.go:117] "RemoveContainer" containerID="e2bb94ca0f8afd062a203f27205716f1518ce2027bf01357ae553842b6c0f69e"
Jul 28 18:18:09.283845 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:18:09.283826    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2bb94ca0f8afd062a203f27205716f1518ce2027bf01357ae553842b6c0f69e\": container with ID starting with e2bb94ca0f8afd062a203f27205716f1518ce2027bf01357ae553842b6c0f69e not found: ID does not exist" containerID="e2bb94ca0f8afd062a203f27205716f1518ce2027bf01357ae553842b6c0f69e"
Jul 28 18:18:09.283880 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:09.283853    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2bb94ca0f8afd062a203f27205716f1518ce2027bf01357ae553842b6c0f69e"} err="failed to get container status \"e2bb94ca0f8afd062a203f27205716f1518ce2027bf01357ae553842b6c0f69e\": rpc error: code = NotFound desc = could not find container \"e2bb94ca0f8afd062a203f27205716f1518ce2027bf01357ae553842b6c0f69e\": container with ID starting with e2bb94ca0f8afd062a203f27205716f1518ce2027bf01357ae553842b6c0f69e not found: ID does not exist"
Jul 28 18:18:09.283880 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:09.283869    2581 scope.go:117] "RemoveContainer" containerID="ce69d9551ecf724f2a41029ecfe19ab7521de155854b99b94f45527ef8aa9ebf"
Jul 28 18:18:09.284197 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:18:09.284182    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce69d9551ecf724f2a41029ecfe19ab7521de155854b99b94f45527ef8aa9ebf\": container with ID starting with ce69d9551ecf724f2a41029ecfe19ab7521de155854b99b94f45527ef8aa9ebf not found: ID does not exist" containerID="ce69d9551ecf724f2a41029ecfe19ab7521de155854b99b94f45527ef8aa9ebf"
Jul 28 18:18:09.284249 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:09.284203    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce69d9551ecf724f2a41029ecfe19ab7521de155854b99b94f45527ef8aa9ebf"} err="failed to get container status \"ce69d9551ecf724f2a41029ecfe19ab7521de155854b99b94f45527ef8aa9ebf\": rpc error: code = NotFound desc = could not find container \"ce69d9551ecf724f2a41029ecfe19ab7521de155854b99b94f45527ef8aa9ebf\": container with ID starting with ce69d9551ecf724f2a41029ecfe19ab7521de155854b99b94f45527ef8aa9ebf not found: ID does not exist"
Jul 28 18:18:09.288396 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:09.288364    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"]
Jul 28 18:18:09.296573 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:09.296550    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-sbjk2"]
Jul 28 18:18:10.289924 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:10.289884    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46e76097-4dbc-426e-bac4-7df016937d30" path="/var/lib/kubelet/pods/46e76097-4dbc-426e-bac4-7df016937d30/volumes"
Jul 28 18:18:16.004328 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:16.004290    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:18:38.352510 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:38.352472    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:18:43.449020 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.448984    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:18:43.505861 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.505825    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"]
Jul 28 18:18:43.506101 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.506088    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46e76097-4dbc-426e-bac4-7df016937d30" containerName="kserve-container"
Jul 28 18:18:43.506173 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.506102    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="46e76097-4dbc-426e-bac4-7df016937d30" containerName="kserve-container"
Jul 28 18:18:43.506173 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.506112    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46e76097-4dbc-426e-bac4-7df016937d30" containerName="kube-rbac-proxy"
Jul 28 18:18:43.506173 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.506118    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="46e76097-4dbc-426e-bac4-7df016937d30" containerName="kube-rbac-proxy"
Jul 28 18:18:43.506173 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.506131    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46e76097-4dbc-426e-bac4-7df016937d30" containerName="storage-initializer"
Jul 28 18:18:43.506173 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.506137    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="46e76097-4dbc-426e-bac4-7df016937d30" containerName="storage-initializer"
Jul 28 18:18:43.506430 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.506177    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="46e76097-4dbc-426e-bac4-7df016937d30" containerName="kserve-container"
Jul 28 18:18:43.506430 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.506186    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="46e76097-4dbc-426e-bac4-7df016937d30" containerName="kube-rbac-proxy"
Jul 28 18:18:43.509167 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.509149    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"
Jul 28 18:18:43.511999 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.511976    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:18:43.512151 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.512049    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 18:18:43.512151 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.512059    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 18:18:43.512294 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.512184    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Jul 28 18:18:43.513103 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.513083    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:18:43.513199 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.513091    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:18:43.513199 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.513159    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:18:43.523384 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.523363    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"]
Jul 28 18:18:43.616467 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.616418    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79013685-840f-46f4-b958-31290752adfd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5\" (UID: \"79013685-840f-46f4-b958-31290752adfd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"
Jul 28 18:18:43.616467 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.616474    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xphwg\" (UniqueName: \"kubernetes.io/projected/79013685-840f-46f4-b958-31290752adfd-kube-api-access-xphwg\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5\" (UID: \"79013685-840f-46f4-b958-31290752adfd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"
Jul 28 18:18:43.616819 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.616528    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79013685-840f-46f4-b958-31290752adfd-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5\" (UID: \"79013685-840f-46f4-b958-31290752adfd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"
Jul 28 18:18:43.616819 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.616553    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79013685-840f-46f4-b958-31290752adfd-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5\" (UID: \"79013685-840f-46f4-b958-31290752adfd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"
Jul 28 18:18:43.616819 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.616639    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79013685-840f-46f4-b958-31290752adfd-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5\" (UID: \"79013685-840f-46f4-b958-31290752adfd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"
Jul 28 18:18:43.717897 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.717782    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79013685-840f-46f4-b958-31290752adfd-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5\" (UID: \"79013685-840f-46f4-b958-31290752adfd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"
Jul 28 18:18:43.717897 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.717873    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79013685-840f-46f4-b958-31290752adfd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5\" (UID: \"79013685-840f-46f4-b958-31290752adfd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"
Jul 28 18:18:43.718183 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.717910    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xphwg\" (UniqueName: \"kubernetes.io/projected/79013685-840f-46f4-b958-31290752adfd-kube-api-access-xphwg\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5\" (UID: \"79013685-840f-46f4-b958-31290752adfd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"
Jul 28 18:18:43.718183 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.717973    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79013685-840f-46f4-b958-31290752adfd-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5\" (UID: \"79013685-840f-46f4-b958-31290752adfd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"
Jul 28 18:18:43.718183 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.718001    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79013685-840f-46f4-b958-31290752adfd-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5\" (UID: \"79013685-840f-46f4-b958-31290752adfd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"
Jul 28 18:18:43.718435 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.718413    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79013685-840f-46f4-b958-31290752adfd-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5\" (UID: \"79013685-840f-46f4-b958-31290752adfd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"
Jul 28 18:18:43.718665 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.718643    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79013685-840f-46f4-b958-31290752adfd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5\" (UID: \"79013685-840f-46f4-b958-31290752adfd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"
Jul 28 18:18:43.718740 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.718697    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79013685-840f-46f4-b958-31290752adfd-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5\" (UID: \"79013685-840f-46f4-b958-31290752adfd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"
Jul 28 18:18:43.720437 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.720412    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79013685-840f-46f4-b958-31290752adfd-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5\" (UID: \"79013685-840f-46f4-b958-31290752adfd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"
Jul 28 18:18:43.720437 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.720424    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:18:43.726381 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.726361    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xphwg\" (UniqueName: \"kubernetes.io/projected/79013685-840f-46f4-b958-31290752adfd-kube-api-access-xphwg\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5\" (UID: \"79013685-840f-46f4-b958-31290752adfd\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"
Jul 28 18:18:43.819155 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.819113    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"
Jul 28 18:18:43.955591 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:43.955558    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"]
Jul 28 18:18:43.958902 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:18:43.958876    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79013685_840f_46f4_b958_31290752adfd.slice/crio-a532a9dcd95fe5afe4c9d20f260b792f3bb929428a0a432f7f094ea247e2d140 WatchSource:0}: Error finding container a532a9dcd95fe5afe4c9d20f260b792f3bb929428a0a432f7f094ea247e2d140: Status 404 returned error can't find the container with id a532a9dcd95fe5afe4c9d20f260b792f3bb929428a0a432f7f094ea247e2d140
Jul 28 18:18:44.379579 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:44.379481    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5" event={"ID":"79013685-840f-46f4-b958-31290752adfd","Type":"ContainerStarted","Data":"77589a5a2a57e6df4a9e62757da7a904eecbb5c015d0ce90f773b93130180c7f"}
Jul 28 18:18:44.379579 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:44.379558    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5" event={"ID":"79013685-840f-46f4-b958-31290752adfd","Type":"ContainerStarted","Data":"a532a9dcd95fe5afe4c9d20f260b792f3bb929428a0a432f7f094ea247e2d140"}
Jul 28 18:18:48.394924 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:48.394889    2581 generic.go:356] "Generic (PLEG): container finished" podID="79013685-840f-46f4-b958-31290752adfd" containerID="77589a5a2a57e6df4a9e62757da7a904eecbb5c015d0ce90f773b93130180c7f" exitCode=0
Jul 28 18:18:48.395350 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:48.394962    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5" event={"ID":"79013685-840f-46f4-b958-31290752adfd","Type":"ContainerDied","Data":"77589a5a2a57e6df4a9e62757da7a904eecbb5c015d0ce90f773b93130180c7f"}
Jul 28 18:18:49.400481 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:49.400443    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5" event={"ID":"79013685-840f-46f4-b958-31290752adfd","Type":"ContainerStarted","Data":"14c5bbdb88ad26ff91b4ff3e75bb91301cdfe81a60e4947f61d8cafc1db59294"}
Jul 28 18:18:49.400882 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:49.400507    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5" event={"ID":"79013685-840f-46f4-b958-31290752adfd","Type":"ContainerStarted","Data":"533ee81bc7aae8b549f42f5040039a2cfd3d72aac8723c76a340366ec1ad68ae"}
Jul 28 18:18:49.400882 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:49.400761    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"
Jul 28 18:18:49.424535 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:49.424482    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5" podStartSLOduration=2.778703611 podStartE2EDuration="6.42446094s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.645757329s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:18:49.422966749 +0000 UTC m=+4177.697583183" watchObservedRunningTime="2026-07-28 18:18:49.42446094 +0000 UTC m=+4177.699077373"
Jul 28 18:18:50.404506 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:50.404476    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"
Jul 28 18:18:56.413265 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:18:56.413228    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"
Jul 28 18:19:12.439323 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:12.439294    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 18:19:12.444639 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:12.444615    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 18:19:26.457954 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:26.457886    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5" podUID="79013685-840f-46f4-b958-31290752adfd" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 28 18:19:36.415899 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:36.415814    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"
Jul 28 18:19:43.629786 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:43.629745    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"]
Jul 28 18:19:43.630364 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:43.630098    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5" podUID="79013685-840f-46f4-b958-31290752adfd" containerName="kserve-container" containerID="cri-o://533ee81bc7aae8b549f42f5040039a2cfd3d72aac8723c76a340366ec1ad68ae" gracePeriod=30
Jul 28 18:19:43.630364 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:43.630175    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5" podUID="79013685-840f-46f4-b958-31290752adfd" containerName="kube-rbac-proxy" containerID="cri-o://14c5bbdb88ad26ff91b4ff3e75bb91301cdfe81a60e4947f61d8cafc1db59294" gracePeriod=30
Jul 28 18:19:43.658307 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:43.658262    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:19:44.596159 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:44.596124    2581 generic.go:356] "Generic (PLEG): container finished" podID="79013685-840f-46f4-b958-31290752adfd" containerID="14c5bbdb88ad26ff91b4ff3e75bb91301cdfe81a60e4947f61d8cafc1db59294" exitCode=2
Jul 28 18:19:44.596336 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:44.596196    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5" event={"ID":"79013685-840f-46f4-b958-31290752adfd","Type":"ContainerDied","Data":"14c5bbdb88ad26ff91b4ff3e75bb91301cdfe81a60e4947f61d8cafc1db59294"}
Jul 28 18:19:46.408505 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:46.408454    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5" podUID="79013685-840f-46f4-b958-31290752adfd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.57:8643/healthz\": dial tcp 10.133.0.57:8643: connect: connection refused"
Jul 28 18:19:48.687708 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:48.687667    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:19:48.719795 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:48.719756    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"]
Jul 28 18:19:48.723268 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:48.723249    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"
Jul 28 18:19:48.725870 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:48.725840    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 28 18:19:48.726125 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:48.726110    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Jul 28 18:19:48.736849 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:48.736828    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"]
Jul 28 18:19:48.904427 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:48.904373    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-c6d677456-s8h2c\" (UID: \"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"
Jul 28 18:19:48.904636 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:48.904454    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-c6d677456-s8h2c\" (UID: \"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"
Jul 28 18:19:48.904636 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:48.904501    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-proxy-tls\") pod \"isvc-sklearn-v2-predictor-c6d677456-s8h2c\" (UID: \"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"
Jul 28 18:19:48.904636 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:48.904529    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-c6d677456-s8h2c\" (UID: \"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"
Jul 28 18:19:48.904636 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:48.904602    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd65d\" (UniqueName: \"kubernetes.io/projected/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-kube-api-access-gd65d\") pod \"isvc-sklearn-v2-predictor-c6d677456-s8h2c\" (UID: \"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"
Jul 28 18:19:49.005988 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:49.005873    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-proxy-tls\") pod \"isvc-sklearn-v2-predictor-c6d677456-s8h2c\" (UID: \"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"
Jul 28 18:19:49.005988 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:49.005920    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-c6d677456-s8h2c\" (UID: \"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"
Jul 28 18:19:49.005988 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:49.005978    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd65d\" (UniqueName: \"kubernetes.io/projected/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-kube-api-access-gd65d\") pod \"isvc-sklearn-v2-predictor-c6d677456-s8h2c\" (UID: \"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"
Jul 28 18:19:49.006254 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:49.006038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-c6d677456-s8h2c\" (UID: \"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"
Jul 28 18:19:49.006254 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:19:49.006065    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
Jul 28 18:19:49.006254 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:49.006080    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-c6d677456-s8h2c\" (UID: \"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"
Jul 28 18:19:49.006254 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:19:49.006136    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-proxy-tls podName:8ab78c13-a9ae-4d81-89a2-5d5fa11ee406 nodeName:}" failed. No retries permitted until 2026-07-28 18:19:49.506111454 +0000 UTC m=+4237.780727869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-proxy-tls") pod "isvc-sklearn-v2-predictor-c6d677456-s8h2c" (UID: "8ab78c13-a9ae-4d81-89a2-5d5fa11ee406") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
Jul 28 18:19:49.006468 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:49.006440    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-c6d677456-s8h2c\" (UID: \"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"
Jul 28 18:19:49.006800 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:49.006782    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-c6d677456-s8h2c\" (UID: \"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"
Jul 28 18:19:49.006848 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:49.006812    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-c6d677456-s8h2c\" (UID: \"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"
Jul 28 18:19:49.015020 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:49.014989    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd65d\" (UniqueName: \"kubernetes.io/projected/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-kube-api-access-gd65d\") pod \"isvc-sklearn-v2-predictor-c6d677456-s8h2c\" (UID: \"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"
Jul 28 18:19:49.108316 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:49.108276    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:19:49.509108 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:49.509075    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-proxy-tls\") pod \"isvc-sklearn-v2-predictor-c6d677456-s8h2c\" (UID: \"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"
Jul 28 18:19:49.511597 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:49.511561    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-proxy-tls\") pod \"isvc-sklearn-v2-predictor-c6d677456-s8h2c\" (UID: \"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"
Jul 28 18:19:49.633496 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:49.633450    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"
Jul 28 18:19:49.766841 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:49.766761    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"]
Jul 28 18:19:49.769616 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:19:49.769580    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab78c13_a9ae_4d81_89a2_5d5fa11ee406.slice/crio-d4ae4a2516de1cad1fa5f550f81cfd3f00d7591f1a19f3f96fb06d391318e92d WatchSource:0}: Error finding container d4ae4a2516de1cad1fa5f550f81cfd3f00d7591f1a19f3f96fb06d391318e92d: Status 404 returned error can't find the container with id d4ae4a2516de1cad1fa5f550f81cfd3f00d7591f1a19f3f96fb06d391318e92d
Jul 28 18:19:50.616890 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:50.616855    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c" event={"ID":"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406","Type":"ContainerStarted","Data":"ee4f2bb5647d28a553645de50f43a793acd26120a899c80514ec335471a968cf"}
Jul 28 18:19:50.617084 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:50.616918    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c" event={"ID":"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406","Type":"ContainerStarted","Data":"d4ae4a2516de1cad1fa5f550f81cfd3f00d7591f1a19f3f96fb06d391318e92d"}
Jul 28 18:19:51.268641 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.268613    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"
Jul 28 18:19:51.326055 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.325960    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79013685-840f-46f4-b958-31290752adfd-cabundle-cert\") pod \"79013685-840f-46f4-b958-31290752adfd\" (UID: \"79013685-840f-46f4-b958-31290752adfd\") "
Jul 28 18:19:51.326055 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.326033    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79013685-840f-46f4-b958-31290752adfd-kserve-provision-location\") pod \"79013685-840f-46f4-b958-31290752adfd\" (UID: \"79013685-840f-46f4-b958-31290752adfd\") "
Jul 28 18:19:51.326055 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.326056    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79013685-840f-46f4-b958-31290752adfd-proxy-tls\") pod \"79013685-840f-46f4-b958-31290752adfd\" (UID: \"79013685-840f-46f4-b958-31290752adfd\") "
Jul 28 18:19:51.326321 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.326079    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xphwg\" (UniqueName: \"kubernetes.io/projected/79013685-840f-46f4-b958-31290752adfd-kube-api-access-xphwg\") pod \"79013685-840f-46f4-b958-31290752adfd\" (UID: \"79013685-840f-46f4-b958-31290752adfd\") "
Jul 28 18:19:51.326321 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.326097    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79013685-840f-46f4-b958-31290752adfd-kube-rbac-proxy-sar-config\") pod \"79013685-840f-46f4-b958-31290752adfd\" (UID: \"79013685-840f-46f4-b958-31290752adfd\") "
Jul 28 18:19:51.326434 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.326386    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79013685-840f-46f4-b958-31290752adfd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "79013685-840f-46f4-b958-31290752adfd" (UID: "79013685-840f-46f4-b958-31290752adfd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:19:51.326490 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.326410    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79013685-840f-46f4-b958-31290752adfd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "79013685-840f-46f4-b958-31290752adfd" (UID: "79013685-840f-46f4-b958-31290752adfd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:19:51.326551 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.326527    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79013685-840f-46f4-b958-31290752adfd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "79013685-840f-46f4-b958-31290752adfd" (UID: "79013685-840f-46f4-b958-31290752adfd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:19:51.328157 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.328136    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79013685-840f-46f4-b958-31290752adfd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "79013685-840f-46f4-b958-31290752adfd" (UID: "79013685-840f-46f4-b958-31290752adfd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:19:51.328238 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.328212    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79013685-840f-46f4-b958-31290752adfd-kube-api-access-xphwg" (OuterVolumeSpecName: "kube-api-access-xphwg") pod "79013685-840f-46f4-b958-31290752adfd" (UID: "79013685-840f-46f4-b958-31290752adfd"). InnerVolumeSpecName "kube-api-access-xphwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:19:51.427022 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.426974    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79013685-840f-46f4-b958-31290752adfd-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:19:51.427022 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.427016    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79013685-840f-46f4-b958-31290752adfd-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:19:51.427022 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.427028    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xphwg\" (UniqueName: \"kubernetes.io/projected/79013685-840f-46f4-b958-31290752adfd-kube-api-access-xphwg\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:19:51.427022 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.427036    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79013685-840f-46f4-b958-31290752adfd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:19:51.427290 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.427045    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79013685-840f-46f4-b958-31290752adfd-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:19:51.623746 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.623658    2581 generic.go:356] "Generic (PLEG): container finished" podID="79013685-840f-46f4-b958-31290752adfd" containerID="533ee81bc7aae8b549f42f5040039a2cfd3d72aac8723c76a340366ec1ad68ae" exitCode=0
Jul 28 18:19:51.623746 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.623729    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"
Jul 28 18:19:51.624089 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.623736    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5" event={"ID":"79013685-840f-46f4-b958-31290752adfd","Type":"ContainerDied","Data":"533ee81bc7aae8b549f42f5040039a2cfd3d72aac8723c76a340366ec1ad68ae"}
Jul 28 18:19:51.624089 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.623810    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5" event={"ID":"79013685-840f-46f4-b958-31290752adfd","Type":"ContainerDied","Data":"a532a9dcd95fe5afe4c9d20f260b792f3bb929428a0a432f7f094ea247e2d140"}
Jul 28 18:19:51.624089 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.623851    2581 scope.go:117] "RemoveContainer" containerID="14c5bbdb88ad26ff91b4ff3e75bb91301cdfe81a60e4947f61d8cafc1db59294"
Jul 28 18:19:51.632695 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.632604    2581 scope.go:117] "RemoveContainer" containerID="533ee81bc7aae8b549f42f5040039a2cfd3d72aac8723c76a340366ec1ad68ae"
Jul 28 18:19:51.640060 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.640041    2581 scope.go:117] "RemoveContainer" containerID="77589a5a2a57e6df4a9e62757da7a904eecbb5c015d0ce90f773b93130180c7f"
Jul 28 18:19:51.646818 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.646797    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"]
Jul 28 18:19:51.647149 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.647125    2581 scope.go:117] "RemoveContainer" containerID="14c5bbdb88ad26ff91b4ff3e75bb91301cdfe81a60e4947f61d8cafc1db59294"
Jul 28 18:19:51.647390 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:19:51.647374    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14c5bbdb88ad26ff91b4ff3e75bb91301cdfe81a60e4947f61d8cafc1db59294\": container with ID starting with 14c5bbdb88ad26ff91b4ff3e75bb91301cdfe81a60e4947f61d8cafc1db59294 not found: ID does not exist" containerID="14c5bbdb88ad26ff91b4ff3e75bb91301cdfe81a60e4947f61d8cafc1db59294"
Jul 28 18:19:51.647447 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.647398    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14c5bbdb88ad26ff91b4ff3e75bb91301cdfe81a60e4947f61d8cafc1db59294"} err="failed to get container status \"14c5bbdb88ad26ff91b4ff3e75bb91301cdfe81a60e4947f61d8cafc1db59294\": rpc error: code = NotFound desc = could not find container \"14c5bbdb88ad26ff91b4ff3e75bb91301cdfe81a60e4947f61d8cafc1db59294\": container with ID starting with 14c5bbdb88ad26ff91b4ff3e75bb91301cdfe81a60e4947f61d8cafc1db59294 not found: ID does not exist"
Jul 28 18:19:51.647447 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.647417    2581 scope.go:117] "RemoveContainer" containerID="533ee81bc7aae8b549f42f5040039a2cfd3d72aac8723c76a340366ec1ad68ae"
Jul 28 18:19:51.647623 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:19:51.647605    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"533ee81bc7aae8b549f42f5040039a2cfd3d72aac8723c76a340366ec1ad68ae\": container with ID starting with 533ee81bc7aae8b549f42f5040039a2cfd3d72aac8723c76a340366ec1ad68ae not found: ID does not exist" containerID="533ee81bc7aae8b549f42f5040039a2cfd3d72aac8723c76a340366ec1ad68ae"
Jul 28 18:19:51.647690 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.647631    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"533ee81bc7aae8b549f42f5040039a2cfd3d72aac8723c76a340366ec1ad68ae"} err="failed to get container status \"533ee81bc7aae8b549f42f5040039a2cfd3d72aac8723c76a340366ec1ad68ae\": rpc error: code = NotFound desc = could not find container \"533ee81bc7aae8b549f42f5040039a2cfd3d72aac8723c76a340366ec1ad68ae\": container with ID starting with 533ee81bc7aae8b549f42f5040039a2cfd3d72aac8723c76a340366ec1ad68ae not found: ID does not exist"
Jul 28 18:19:51.647690 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.647658    2581 scope.go:117] "RemoveContainer" containerID="77589a5a2a57e6df4a9e62757da7a904eecbb5c015d0ce90f773b93130180c7f"
Jul 28 18:19:51.647863 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:19:51.647846    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77589a5a2a57e6df4a9e62757da7a904eecbb5c015d0ce90f773b93130180c7f\": container with ID starting with 77589a5a2a57e6df4a9e62757da7a904eecbb5c015d0ce90f773b93130180c7f not found: ID does not exist" containerID="77589a5a2a57e6df4a9e62757da7a904eecbb5c015d0ce90f773b93130180c7f"
Jul 28 18:19:51.647904 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.647866    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77589a5a2a57e6df4a9e62757da7a904eecbb5c015d0ce90f773b93130180c7f"} err="failed to get container status \"77589a5a2a57e6df4a9e62757da7a904eecbb5c015d0ce90f773b93130180c7f\": rpc error: code = NotFound desc = could not find container \"77589a5a2a57e6df4a9e62757da7a904eecbb5c015d0ce90f773b93130180c7f\": container with ID starting with 77589a5a2a57e6df4a9e62757da7a904eecbb5c015d0ce90f773b93130180c7f not found: ID does not exist"
Jul 28 18:19:51.650987 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:51.650966    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-strw5"]
Jul 28 18:19:52.289163 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:52.289131    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79013685-840f-46f4-b958-31290752adfd" path="/var/lib/kubelet/pods/79013685-840f-46f4-b958-31290752adfd/volumes"
Jul 28 18:19:53.632586 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:53.632548    2581 generic.go:356] "Generic (PLEG): container finished" podID="8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" containerID="ee4f2bb5647d28a553645de50f43a793acd26120a899c80514ec335471a968cf" exitCode=0
Jul 28 18:19:53.632962 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:53.632621    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c" event={"ID":"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406","Type":"ContainerDied","Data":"ee4f2bb5647d28a553645de50f43a793acd26120a899c80514ec335471a968cf"}
Jul 28 18:19:54.637998 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:54.637937    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c" event={"ID":"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406","Type":"ContainerStarted","Data":"6d1cbdd6060f1516004d56524de7cdbf126446eb071f9bf9b063d290056bb50e"}
Jul 28 18:19:54.638424 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:54.638027    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c" event={"ID":"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406","Type":"ContainerStarted","Data":"3c907bb7503c04df5a4c84cd8e4035e17dfe223d8686db8adb706f4b519c6c3e"}
Jul 28 18:19:54.638424 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:54.638229    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"
Jul 28 18:19:54.659112 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:54.659065    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c" podStartSLOduration=3.078043204 podStartE2EDuration="6.65904944s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.581006236s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:19:54.657447345 +0000 UTC m=+4242.932063776" watchObservedRunningTime="2026-07-28 18:19:54.65904944 +0000 UTC m=+4242.933665872"
Jul 28 18:19:55.642081 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:55.642048    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"
Jul 28 18:19:55.643431 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:55.643401    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c" podUID="8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 18:19:56.645110 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:19:56.645063    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c" podUID="8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 18:20:01.649571 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:20:01.649536    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"
Jul 28 18:20:01.650204 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:20:01.650174    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c" podUID="8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 18:20:11.650462 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:20:11.650417    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c" podUID="8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 18:20:21.650416 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:20:21.650375    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c" podUID="8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 18:20:31.651014 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:20:31.650972    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c" podUID="8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 18:20:41.650299 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:20:41.650255    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c" podUID="8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 18:20:51.650680 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:20:51.650637    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c" podUID="8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 18:21:01.651024 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:01.650981    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c" podUID="8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 18:21:11.651078 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:11.650997    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"
Jul 28 18:21:18.939015 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:18.938980    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"]
Jul 28 18:21:18.941527 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:18.939240    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c" podUID="8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" containerName="kserve-container" containerID="cri-o://3c907bb7503c04df5a4c84cd8e4035e17dfe223d8686db8adb706f4b519c6c3e" gracePeriod=30
Jul 28 18:21:18.941527 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:18.939291    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c" podUID="8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" containerName="kube-rbac-proxy" containerID="cri-o://6d1cbdd6060f1516004d56524de7cdbf126446eb071f9bf9b063d290056bb50e" gracePeriod=30
Jul 28 18:21:18.966127 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:18.966088    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:21:19.931881 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:19.931843    2581 generic.go:356] "Generic (PLEG): container finished" podID="8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" containerID="6d1cbdd6060f1516004d56524de7cdbf126446eb071f9bf9b063d290056bb50e" exitCode=2
Jul 28 18:21:19.932085 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:19.931901    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c" event={"ID":"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406","Type":"ContainerDied","Data":"6d1cbdd6060f1516004d56524de7cdbf126446eb071f9bf9b063d290056bb50e"}
Jul 28 18:21:21.645966 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:21.645893    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c" podUID="8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.58:8643/healthz\": dial tcp 10.133.0.58:8643: connect: connection refused"
Jul 28 18:21:21.650198 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:21.650172    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c" podUID="8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Jul 28 18:21:23.948379 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:23.948339    2581 generic.go:356] "Generic (PLEG): container finished" podID="8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" containerID="3c907bb7503c04df5a4c84cd8e4035e17dfe223d8686db8adb706f4b519c6c3e" exitCode=0
Jul 28 18:21:23.948749 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:23.948417    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c" event={"ID":"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406","Type":"ContainerDied","Data":"3c907bb7503c04df5a4c84cd8e4035e17dfe223d8686db8adb706f4b519c6c3e"}
Jul 28 18:21:23.987123 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:23.987098    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"
Jul 28 18:21:24.032825 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.032739    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gd65d\" (UniqueName: \"kubernetes.io/projected/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-kube-api-access-gd65d\") pod \"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406\" (UID: \"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406\") "
Jul 28 18:21:24.032825 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.032788    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-kube-rbac-proxy-sar-config\") pod \"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406\" (UID: \"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406\") "
Jul 28 18:21:24.033074 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.032828    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-kserve-provision-location\") pod \"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406\" (UID: \"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406\") "
Jul 28 18:21:24.033074 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.032956    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-cabundle-cert\") pod \"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406\" (UID: \"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406\") "
Jul 28 18:21:24.033074 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.033037    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-proxy-tls\") pod \"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406\" (UID: \"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406\") "
Jul 28 18:21:24.033250 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.033140    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" (UID: "8ab78c13-a9ae-4d81-89a2-5d5fa11ee406"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:21:24.033308 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.033257    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" (UID: "8ab78c13-a9ae-4d81-89a2-5d5fa11ee406"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:21:24.033308 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.033282    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" (UID: "8ab78c13-a9ae-4d81-89a2-5d5fa11ee406"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:21:24.034889 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.034865    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-kube-api-access-gd65d" (OuterVolumeSpecName: "kube-api-access-gd65d") pod "8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" (UID: "8ab78c13-a9ae-4d81-89a2-5d5fa11ee406"). InnerVolumeSpecName "kube-api-access-gd65d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:21:24.035105 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.034967    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" (UID: "8ab78c13-a9ae-4d81-89a2-5d5fa11ee406"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:21:24.048607 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.048573    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:21:24.110268 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.110228    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"]
Jul 28 18:21:24.110486 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.110475    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79013685-840f-46f4-b958-31290752adfd" containerName="storage-initializer"
Jul 28 18:21:24.110529 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.110488    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="79013685-840f-46f4-b958-31290752adfd" containerName="storage-initializer"
Jul 28 18:21:24.110529 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.110498    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79013685-840f-46f4-b958-31290752adfd" containerName="kserve-container"
Jul 28 18:21:24.110529 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.110504    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="79013685-840f-46f4-b958-31290752adfd" containerName="kserve-container"
Jul 28 18:21:24.110529 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.110512    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" containerName="kube-rbac-proxy"
Jul 28 18:21:24.110529 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.110518    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" containerName="kube-rbac-proxy"
Jul 28 18:21:24.110529 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.110528    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" containerName="kserve-container"
Jul 28 18:21:24.110732 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.110533    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" containerName="kserve-container"
Jul 28 18:21:24.110732 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.110542    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79013685-840f-46f4-b958-31290752adfd" containerName="kube-rbac-proxy"
Jul 28 18:21:24.110732 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.110547    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="79013685-840f-46f4-b958-31290752adfd" containerName="kube-rbac-proxy"
Jul 28 18:21:24.110732 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.110555    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" containerName="storage-initializer"
Jul 28 18:21:24.110732 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.110560    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" containerName="storage-initializer"
Jul 28 18:21:24.110732 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.110600    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="79013685-840f-46f4-b958-31290752adfd" containerName="kube-rbac-proxy"
Jul 28 18:21:24.110732 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.110608    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" containerName="kserve-container"
Jul 28 18:21:24.110732 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.110615    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" containerName="kube-rbac-proxy"
Jul 28 18:21:24.110732 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.110621    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="79013685-840f-46f4-b958-31290752adfd" containerName="kserve-container"
Jul 28 18:21:24.113729 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.113710    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"
Jul 28 18:21:24.116647 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.116616    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Jul 28 18:21:24.116791 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.116629    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Jul 28 18:21:24.125058 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.125037    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"]
Jul 28 18:21:24.133933 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.133901    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c880d52-efbb-4038-9e44-44c0f2aca81a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh\" (UID: \"4c880d52-efbb-4038-9e44-44c0f2aca81a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"
Jul 28 18:21:24.134095 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.133965    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c880d52-efbb-4038-9e44-44c0f2aca81a-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh\" (UID: \"4c880d52-efbb-4038-9e44-44c0f2aca81a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"
Jul 28 18:21:24.134095 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.134005    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c880d52-efbb-4038-9e44-44c0f2aca81a-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh\" (UID: \"4c880d52-efbb-4038-9e44-44c0f2aca81a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"
Jul 28 18:21:24.134095 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.134028    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7hbm\" (UniqueName: \"kubernetes.io/projected/4c880d52-efbb-4038-9e44-44c0f2aca81a-kube-api-access-q7hbm\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh\" (UID: \"4c880d52-efbb-4038-9e44-44c0f2aca81a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"
Jul 28 18:21:24.134095 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.134064    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c880d52-efbb-4038-9e44-44c0f2aca81a-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh\" (UID: \"4c880d52-efbb-4038-9e44-44c0f2aca81a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"
Jul 28 18:21:24.134235 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.134099    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:21:24.134235 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.134116    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd65d\" (UniqueName: \"kubernetes.io/projected/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-kube-api-access-gd65d\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:21:24.134235 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.134129    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:21:24.134235 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.134138    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:21:24.134235 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.134174    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:21:24.150962 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.150910    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:21:24.234803 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.234766    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c880d52-efbb-4038-9e44-44c0f2aca81a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh\" (UID: \"4c880d52-efbb-4038-9e44-44c0f2aca81a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"
Jul 28 18:21:24.235004 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.234810    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c880d52-efbb-4038-9e44-44c0f2aca81a-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh\" (UID: \"4c880d52-efbb-4038-9e44-44c0f2aca81a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"
Jul 28 18:21:24.235004 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.234932    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c880d52-efbb-4038-9e44-44c0f2aca81a-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh\" (UID: \"4c880d52-efbb-4038-9e44-44c0f2aca81a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"
Jul 28 18:21:24.235118 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.235004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7hbm\" (UniqueName: \"kubernetes.io/projected/4c880d52-efbb-4038-9e44-44c0f2aca81a-kube-api-access-q7hbm\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh\" (UID: \"4c880d52-efbb-4038-9e44-44c0f2aca81a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"
Jul 28 18:21:24.235118 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.235051    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c880d52-efbb-4038-9e44-44c0f2aca81a-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh\" (UID: \"4c880d52-efbb-4038-9e44-44c0f2aca81a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"
Jul 28 18:21:24.235407 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.235375    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c880d52-efbb-4038-9e44-44c0f2aca81a-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh\" (UID: \"4c880d52-efbb-4038-9e44-44c0f2aca81a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"
Jul 28 18:21:24.235560 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.235542    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c880d52-efbb-4038-9e44-44c0f2aca81a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh\" (UID: \"4c880d52-efbb-4038-9e44-44c0f2aca81a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"
Jul 28 18:21:24.235608 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.235558    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c880d52-efbb-4038-9e44-44c0f2aca81a-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh\" (UID: \"4c880d52-efbb-4038-9e44-44c0f2aca81a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"
Jul 28 18:21:24.237375 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.237358    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c880d52-efbb-4038-9e44-44c0f2aca81a-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh\" (UID: \"4c880d52-efbb-4038-9e44-44c0f2aca81a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"
Jul 28 18:21:24.244173 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.244150    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7hbm\" (UniqueName: \"kubernetes.io/projected/4c880d52-efbb-4038-9e44-44c0f2aca81a-kube-api-access-q7hbm\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh\" (UID: \"4c880d52-efbb-4038-9e44-44c0f2aca81a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"
Jul 28 18:21:24.423132 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.423093    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"
Jul 28 18:21:24.553702 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.553483    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"]
Jul 28 18:21:24.556448 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:21:24.556421    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c880d52_efbb_4038_9e44_44c0f2aca81a.slice/crio-d309da8369c64bbf79b01f51f8ce7b9e5fed0a4d982aa7b60ee29cd0d512c69f WatchSource:0}: Error finding container d309da8369c64bbf79b01f51f8ce7b9e5fed0a4d982aa7b60ee29cd0d512c69f: Status 404 returned error can't find the container with id d309da8369c64bbf79b01f51f8ce7b9e5fed0a4d982aa7b60ee29cd0d512c69f
Jul 28 18:21:24.953490 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.953451    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh" event={"ID":"4c880d52-efbb-4038-9e44-44c0f2aca81a","Type":"ContainerStarted","Data":"0387a3d536b1a69b54bce5a96312d35614cb3b88f3e271239ba1eed0dcea5b6d"}
Jul 28 18:21:24.953970 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.953532    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh" event={"ID":"4c880d52-efbb-4038-9e44-44c0f2aca81a","Type":"ContainerStarted","Data":"d309da8369c64bbf79b01f51f8ce7b9e5fed0a4d982aa7b60ee29cd0d512c69f"}
Jul 28 18:21:24.955421 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.955392    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c" event={"ID":"8ab78c13-a9ae-4d81-89a2-5d5fa11ee406","Type":"ContainerDied","Data":"d4ae4a2516de1cad1fa5f550f81cfd3f00d7591f1a19f3f96fb06d391318e92d"}
Jul 28 18:21:24.955535 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.955458    2581 scope.go:117] "RemoveContainer" containerID="6d1cbdd6060f1516004d56524de7cdbf126446eb071f9bf9b063d290056bb50e"
Jul 28 18:21:24.955535 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.955398    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"
Jul 28 18:21:24.964515 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.964495    2581 scope.go:117] "RemoveContainer" containerID="3c907bb7503c04df5a4c84cd8e4035e17dfe223d8686db8adb706f4b519c6c3e"
Jul 28 18:21:24.971485 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.971466    2581 scope.go:117] "RemoveContainer" containerID="ee4f2bb5647d28a553645de50f43a793acd26120a899c80514ec335471a968cf"
Jul 28 18:21:24.990536 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.990506    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"]
Jul 28 18:21:24.995397 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:24.995362    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-c6d677456-s8h2c"]
Jul 28 18:21:26.290021 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:26.289990    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ab78c13-a9ae-4d81-89a2-5d5fa11ee406" path="/var/lib/kubelet/pods/8ab78c13-a9ae-4d81-89a2-5d5fa11ee406/volumes"
Jul 28 18:21:28.974389 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:28.974356    2581 generic.go:356] "Generic (PLEG): container finished" podID="4c880d52-efbb-4038-9e44-44c0f2aca81a" containerID="0387a3d536b1a69b54bce5a96312d35614cb3b88f3e271239ba1eed0dcea5b6d" exitCode=0
Jul 28 18:21:28.974751 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:28.974415    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh" event={"ID":"4c880d52-efbb-4038-9e44-44c0f2aca81a","Type":"ContainerDied","Data":"0387a3d536b1a69b54bce5a96312d35614cb3b88f3e271239ba1eed0dcea5b6d"}
Jul 28 18:21:29.979544 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:29.979504    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh" event={"ID":"4c880d52-efbb-4038-9e44-44c0f2aca81a","Type":"ContainerStarted","Data":"83953b12c732c5874eb624c64c433cc741325945eb9efbb5a40cefe92eda42b9"}
Jul 28 18:21:29.979929 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:29.979569    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh" event={"ID":"4c880d52-efbb-4038-9e44-44c0f2aca81a","Type":"ContainerStarted","Data":"a256d0ebd771b817748ed0f54a965c83ba4aff3ec59f3b3a43b34aa51a938b41"}
Jul 28 18:21:29.979929 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:29.979862    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"
Jul 28 18:21:29.980026 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:29.980004    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"
Jul 28 18:21:29.981408 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:29.981380    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh" podUID="4c880d52-efbb-4038-9e44-44c0f2aca81a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 18:21:30.000321 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:30.000277    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh" podStartSLOduration=2.009657585 podStartE2EDuration="6.000260504s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.990602919s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:21:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:21:29.998871097 +0000 UTC m=+4338.273487529" watchObservedRunningTime="2026-07-28 18:21:30.000260504 +0000 UTC m=+4338.274876935"
Jul 28 18:21:30.983341 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:30.983298    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh" podUID="4c880d52-efbb-4038-9e44-44c0f2aca81a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 18:21:35.988315 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:35.988282    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"
Jul 28 18:21:35.988976 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:35.988931    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh" podUID="4c880d52-efbb-4038-9e44-44c0f2aca81a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 18:21:45.989402 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:45.989361    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh" podUID="4c880d52-efbb-4038-9e44-44c0f2aca81a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 18:21:53.070134 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:53.070083    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:21:55.989466 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:21:55.989417    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh" podUID="4c880d52-efbb-4038-9e44-44c0f2aca81a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 18:22:05.989776 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:05.989731    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh" podUID="4c880d52-efbb-4038-9e44-44c0f2aca81a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 18:22:15.989692 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:15.989648    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh" podUID="4c880d52-efbb-4038-9e44-44c0f2aca81a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 18:22:25.989552 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:25.989509    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh" podUID="4c880d52-efbb-4038-9e44-44c0f2aca81a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 18:22:35.989750 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:35.989663    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh" podUID="4c880d52-efbb-4038-9e44-44c0f2aca81a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 18:22:45.989778 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:45.989743    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"
Jul 28 18:22:51.050261 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:51.050218    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:22:54.372849 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:54.372811    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"]
Jul 28 18:22:54.373251 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:54.373132    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh" podUID="4c880d52-efbb-4038-9e44-44c0f2aca81a" containerName="kserve-container" containerID="cri-o://a256d0ebd771b817748ed0f54a965c83ba4aff3ec59f3b3a43b34aa51a938b41" gracePeriod=30
Jul 28 18:22:54.373251 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:54.373155    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh" podUID="4c880d52-efbb-4038-9e44-44c0f2aca81a" containerName="kube-rbac-proxy" containerID="cri-o://83953b12c732c5874eb624c64c433cc741325945eb9efbb5a40cefe92eda42b9" gracePeriod=30
Jul 28 18:22:54.397344 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:54.397309    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:22:55.276199 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:55.276162    2581 generic.go:356] "Generic (PLEG): container finished" podID="4c880d52-efbb-4038-9e44-44c0f2aca81a" containerID="83953b12c732c5874eb624c64c433cc741325945eb9efbb5a40cefe92eda42b9" exitCode=2
Jul 28 18:22:55.276384 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:55.276205    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh" event={"ID":"4c880d52-efbb-4038-9e44-44c0f2aca81a","Type":"ContainerDied","Data":"83953b12c732c5874eb624c64c433cc741325945eb9efbb5a40cefe92eda42b9"}
Jul 28 18:22:55.984022 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:55.983971    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh" podUID="4c880d52-efbb-4038-9e44-44c0f2aca81a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.59:8643/healthz\": dial tcp 10.133.0.59:8643: connect: connection refused"
Jul 28 18:22:55.989334 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:55.989305    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh" podUID="4c880d52-efbb-4038-9e44-44c0f2aca81a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Jul 28 18:22:59.293289 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.293246    2581 generic.go:356] "Generic (PLEG): container finished" podID="4c880d52-efbb-4038-9e44-44c0f2aca81a" containerID="a256d0ebd771b817748ed0f54a965c83ba4aff3ec59f3b3a43b34aa51a938b41" exitCode=0
Jul 28 18:22:59.293289 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.293291    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh" event={"ID":"4c880d52-efbb-4038-9e44-44c0f2aca81a","Type":"ContainerDied","Data":"a256d0ebd771b817748ed0f54a965c83ba4aff3ec59f3b3a43b34aa51a938b41"}
Jul 28 18:22:59.324854 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.324815    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"
Jul 28 18:22:59.343978 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.343932    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c880d52-efbb-4038-9e44-44c0f2aca81a-kube-rbac-proxy-sar-config\") pod \"4c880d52-efbb-4038-9e44-44c0f2aca81a\" (UID: \"4c880d52-efbb-4038-9e44-44c0f2aca81a\") "
Jul 28 18:22:59.343978 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.343983    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c880d52-efbb-4038-9e44-44c0f2aca81a-proxy-tls\") pod \"4c880d52-efbb-4038-9e44-44c0f2aca81a\" (UID: \"4c880d52-efbb-4038-9e44-44c0f2aca81a\") "
Jul 28 18:22:59.344217 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.344005    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c880d52-efbb-4038-9e44-44c0f2aca81a-kserve-provision-location\") pod \"4c880d52-efbb-4038-9e44-44c0f2aca81a\" (UID: \"4c880d52-efbb-4038-9e44-44c0f2aca81a\") "
Jul 28 18:22:59.344217 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.344054    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c880d52-efbb-4038-9e44-44c0f2aca81a-cabundle-cert\") pod \"4c880d52-efbb-4038-9e44-44c0f2aca81a\" (UID: \"4c880d52-efbb-4038-9e44-44c0f2aca81a\") "
Jul 28 18:22:59.344370 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.344330    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c880d52-efbb-4038-9e44-44c0f2aca81a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4c880d52-efbb-4038-9e44-44c0f2aca81a" (UID: "4c880d52-efbb-4038-9e44-44c0f2aca81a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:22:59.344496 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.344471    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c880d52-efbb-4038-9e44-44c0f2aca81a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4c880d52-efbb-4038-9e44-44c0f2aca81a" (UID: "4c880d52-efbb-4038-9e44-44c0f2aca81a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:22:59.344496 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.344488    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c880d52-efbb-4038-9e44-44c0f2aca81a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4c880d52-efbb-4038-9e44-44c0f2aca81a" (UID: "4c880d52-efbb-4038-9e44-44c0f2aca81a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:22:59.346113 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.346092    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c880d52-efbb-4038-9e44-44c0f2aca81a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4c880d52-efbb-4038-9e44-44c0f2aca81a" (UID: "4c880d52-efbb-4038-9e44-44c0f2aca81a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:22:59.355816 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.355784    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:22:59.429829 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.429720    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"]
Jul 28 18:22:59.430104 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.430088    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c880d52-efbb-4038-9e44-44c0f2aca81a" containerName="storage-initializer"
Jul 28 18:22:59.430157 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.430107    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c880d52-efbb-4038-9e44-44c0f2aca81a" containerName="storage-initializer"
Jul 28 18:22:59.430157 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.430121    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c880d52-efbb-4038-9e44-44c0f2aca81a" containerName="kserve-container"
Jul 28 18:22:59.430157 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.430129    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c880d52-efbb-4038-9e44-44c0f2aca81a" containerName="kserve-container"
Jul 28 18:22:59.430316 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.430159    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c880d52-efbb-4038-9e44-44c0f2aca81a" containerName="kube-rbac-proxy"
Jul 28 18:22:59.430316 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.430168    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c880d52-efbb-4038-9e44-44c0f2aca81a" containerName="kube-rbac-proxy"
Jul 28 18:22:59.430316 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.430224    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c880d52-efbb-4038-9e44-44c0f2aca81a" containerName="kube-rbac-proxy"
Jul 28 18:22:59.430316 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.430237    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c880d52-efbb-4038-9e44-44c0f2aca81a" containerName="kserve-container"
Jul 28 18:22:59.433759 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.433734    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"
Jul 28 18:22:59.436562 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.436532    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Jul 28 18:22:59.436709 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.436535    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Jul 28 18:22:59.445074 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.445053    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7hbm\" (UniqueName: \"kubernetes.io/projected/4c880d52-efbb-4038-9e44-44c0f2aca81a-kube-api-access-q7hbm\") pod \"4c880d52-efbb-4038-9e44-44c0f2aca81a\" (UID: \"4c880d52-efbb-4038-9e44-44c0f2aca81a\") "
Jul 28 18:22:59.445208 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.445158    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3636a612-4b85-4aef-9143-39a4de422984-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-km2gc\" (UID: \"3636a612-4b85-4aef-9143-39a4de422984\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"
Jul 28 18:22:59.445277 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.445214    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpdgt\" (UniqueName: \"kubernetes.io/projected/3636a612-4b85-4aef-9143-39a4de422984-kube-api-access-lpdgt\") pod \"isvc-tensorflow-predictor-547dfc5669-km2gc\" (UID: \"3636a612-4b85-4aef-9143-39a4de422984\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"
Jul 28 18:22:59.445277 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.445242    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3636a612-4b85-4aef-9143-39a4de422984-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-km2gc\" (UID: \"3636a612-4b85-4aef-9143-39a4de422984\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"
Jul 28 18:22:59.445277 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.445272    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3636a612-4b85-4aef-9143-39a4de422984-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-km2gc\" (UID: \"3636a612-4b85-4aef-9143-39a4de422984\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"
Jul 28 18:22:59.445443 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.445315    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3636a612-4b85-4aef-9143-39a4de422984-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-km2gc\" (UID: \"3636a612-4b85-4aef-9143-39a4de422984\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"
Jul 28 18:22:59.445443 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.445357    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c880d52-efbb-4038-9e44-44c0f2aca81a-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:22:59.445443 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.445379    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"]
Jul 28 18:22:59.445443 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.445393    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c880d52-efbb-4038-9e44-44c0f2aca81a-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:22:59.445443 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.445409    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c880d52-efbb-4038-9e44-44c0f2aca81a-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:22:59.445443 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.445423    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c880d52-efbb-4038-9e44-44c0f2aca81a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:22:59.447092 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.447070    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c880d52-efbb-4038-9e44-44c0f2aca81a-kube-api-access-q7hbm" (OuterVolumeSpecName: "kube-api-access-q7hbm") pod "4c880d52-efbb-4038-9e44-44c0f2aca81a" (UID: "4c880d52-efbb-4038-9e44-44c0f2aca81a"). InnerVolumeSpecName "kube-api-access-q7hbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:22:59.546498 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.546460    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3636a612-4b85-4aef-9143-39a4de422984-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-km2gc\" (UID: \"3636a612-4b85-4aef-9143-39a4de422984\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"
Jul 28 18:22:59.546705 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.546507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpdgt\" (UniqueName: \"kubernetes.io/projected/3636a612-4b85-4aef-9143-39a4de422984-kube-api-access-lpdgt\") pod \"isvc-tensorflow-predictor-547dfc5669-km2gc\" (UID: \"3636a612-4b85-4aef-9143-39a4de422984\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"
Jul 28 18:22:59.546705 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.546525    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3636a612-4b85-4aef-9143-39a4de422984-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-km2gc\" (UID: \"3636a612-4b85-4aef-9143-39a4de422984\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"
Jul 28 18:22:59.546705 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.546545    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3636a612-4b85-4aef-9143-39a4de422984-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-km2gc\" (UID: \"3636a612-4b85-4aef-9143-39a4de422984\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"
Jul 28 18:22:59.546705 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.546570    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3636a612-4b85-4aef-9143-39a4de422984-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-km2gc\" (UID: \"3636a612-4b85-4aef-9143-39a4de422984\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"
Jul 28 18:22:59.546705 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.546612    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7hbm\" (UniqueName: \"kubernetes.io/projected/4c880d52-efbb-4038-9e44-44c0f2aca81a-kube-api-access-q7hbm\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:22:59.547013 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.546977    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3636a612-4b85-4aef-9143-39a4de422984-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-km2gc\" (UID: \"3636a612-4b85-4aef-9143-39a4de422984\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"
Jul 28 18:22:59.547206 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.547188    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3636a612-4b85-4aef-9143-39a4de422984-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-km2gc\" (UID: \"3636a612-4b85-4aef-9143-39a4de422984\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"
Jul 28 18:22:59.547282 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.547265    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3636a612-4b85-4aef-9143-39a4de422984-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-km2gc\" (UID: \"3636a612-4b85-4aef-9143-39a4de422984\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"
Jul 28 18:22:59.549057 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.549036    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3636a612-4b85-4aef-9143-39a4de422984-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-km2gc\" (UID: \"3636a612-4b85-4aef-9143-39a4de422984\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"
Jul 28 18:22:59.554989 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.554959    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:22:59.573258 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.573234    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpdgt\" (UniqueName: \"kubernetes.io/projected/3636a612-4b85-4aef-9143-39a4de422984-kube-api-access-lpdgt\") pod \"isvc-tensorflow-predictor-547dfc5669-km2gc\" (UID: \"3636a612-4b85-4aef-9143-39a4de422984\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"
Jul 28 18:22:59.746451 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.746363    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"
Jul 28 18:22:59.874101 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:22:59.874066    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"]
Jul 28 18:22:59.877423 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:22:59.877390    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3636a612_4b85_4aef_9143_39a4de422984.slice/crio-9c1ad8e1a884e54dad7036b2fa523398224fcded09363df942f7ed84153de690 WatchSource:0}: Error finding container 9c1ad8e1a884e54dad7036b2fa523398224fcded09363df942f7ed84153de690: Status 404 returned error can't find the container with id 9c1ad8e1a884e54dad7036b2fa523398224fcded09363df942f7ed84153de690
Jul 28 18:23:00.297807 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:00.297764    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc" event={"ID":"3636a612-4b85-4aef-9143-39a4de422984","Type":"ContainerStarted","Data":"399995b214a451c86d9b60b65c374cc267119888befdd4f713472005443bb0be"}
Jul 28 18:23:00.298321 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:00.297837    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc" event={"ID":"3636a612-4b85-4aef-9143-39a4de422984","Type":"ContainerStarted","Data":"9c1ad8e1a884e54dad7036b2fa523398224fcded09363df942f7ed84153de690"}
Jul 28 18:23:00.299740 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:00.299712    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh" event={"ID":"4c880d52-efbb-4038-9e44-44c0f2aca81a","Type":"ContainerDied","Data":"d309da8369c64bbf79b01f51f8ce7b9e5fed0a4d982aa7b60ee29cd0d512c69f"}
Jul 28 18:23:00.299883 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:00.299772    2581 scope.go:117] "RemoveContainer" containerID="83953b12c732c5874eb624c64c433cc741325945eb9efbb5a40cefe92eda42b9"
Jul 28 18:23:00.299883 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:00.299817    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"
Jul 28 18:23:00.312021 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:00.311990    2581 scope.go:117] "RemoveContainer" containerID="a256d0ebd771b817748ed0f54a965c83ba4aff3ec59f3b3a43b34aa51a938b41"
Jul 28 18:23:00.321686 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:00.321665    2581 scope.go:117] "RemoveContainer" containerID="0387a3d536b1a69b54bce5a96312d35614cb3b88f3e271239ba1eed0dcea5b6d"
Jul 28 18:23:00.331653 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:00.331617    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"]
Jul 28 18:23:00.338077 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:00.338045    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-lc6nh"]
Jul 28 18:23:02.289981 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:02.289928    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c880d52-efbb-4038-9e44-44c0f2aca81a" path="/var/lib/kubelet/pods/4c880d52-efbb-4038-9e44-44c0f2aca81a/volumes"
Jul 28 18:23:05.318234 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:05.318196    2581 generic.go:356] "Generic (PLEG): container finished" podID="3636a612-4b85-4aef-9143-39a4de422984" containerID="399995b214a451c86d9b60b65c374cc267119888befdd4f713472005443bb0be" exitCode=0
Jul 28 18:23:05.318234 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:05.318237    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc" event={"ID":"3636a612-4b85-4aef-9143-39a4de422984","Type":"ContainerDied","Data":"399995b214a451c86d9b60b65c374cc267119888befdd4f713472005443bb0be"}
Jul 28 18:23:05.319560 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:05.319541    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:23:09.341305 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:09.341215    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc" event={"ID":"3636a612-4b85-4aef-9143-39a4de422984","Type":"ContainerStarted","Data":"f0999cd22603793201dea5422da610bfebcd0a2ee4f91724128bf1fcff22dc18"}
Jul 28 18:23:09.341305 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:09.341299    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc" event={"ID":"3636a612-4b85-4aef-9143-39a4de422984","Type":"ContainerStarted","Data":"060c262bf7c156d0090b5de69fc06e6c217d2bd6b7f911fb12cb6379485c3953"}
Jul 28 18:23:09.341744 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:09.341486    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"
Jul 28 18:23:09.365025 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:09.364962    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc" podStartSLOduration=2.316377626 podStartE2EDuration="10.364926382s" totalImagesPullingTime="3.655352448s" totalInitContainerRuntime="4.393196308s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:22:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:23:09.362913904 +0000 UTC m=+4437.637530336" watchObservedRunningTime="2026-07-28 18:23:09.364926382 +0000 UTC m=+4437.639542814"
Jul 28 18:23:10.344841 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:10.344810    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"
Jul 28 18:23:10.346162 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:10.346133    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc" podUID="3636a612-4b85-4aef-9143-39a4de422984" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 28 18:23:11.348926 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:11.348879    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc" podUID="3636a612-4b85-4aef-9143-39a4de422984" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 28 18:23:16.355433 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:16.355399    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"
Jul 28 18:23:16.356005 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:16.355972    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc" podUID="3636a612-4b85-4aef-9143-39a4de422984" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 28 18:23:26.356718 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:26.356688    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"
Jul 28 18:23:40.930688 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:40.930638    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"]
Jul 28 18:23:40.931214 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:40.930915    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc" podUID="3636a612-4b85-4aef-9143-39a4de422984" containerName="kserve-container" containerID="cri-o://060c262bf7c156d0090b5de69fc06e6c217d2bd6b7f911fb12cb6379485c3953" gracePeriod=30
Jul 28 18:23:40.931214 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:40.931018    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc" podUID="3636a612-4b85-4aef-9143-39a4de422984" containerName="kube-rbac-proxy" containerID="cri-o://f0999cd22603793201dea5422da610bfebcd0a2ee4f91724128bf1fcff22dc18" gracePeriod=30
Jul 28 18:23:40.959628 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:40.959593    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:23:41.349573 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:41.349478    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc" podUID="3636a612-4b85-4aef-9143-39a4de422984" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.60:8643/healthz\": dial tcp 10.133.0.60:8643: connect: connection refused"
Jul 28 18:23:41.455920 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:41.455885    2581 generic.go:356] "Generic (PLEG): container finished" podID="3636a612-4b85-4aef-9143-39a4de422984" containerID="f0999cd22603793201dea5422da610bfebcd0a2ee4f91724128bf1fcff22dc18" exitCode=2
Jul 28 18:23:41.456109 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:41.455927    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc" event={"ID":"3636a612-4b85-4aef-9143-39a4de422984","Type":"ContainerDied","Data":"f0999cd22603793201dea5422da610bfebcd0a2ee4f91724128bf1fcff22dc18"}
Jul 28 18:23:46.084071 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:46.084037    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:23:46.120249 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:46.120212    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"]
Jul 28 18:23:46.123550 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:46.123533    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"
Jul 28 18:23:46.126049 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:46.126021    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Jul 28 18:23:46.126177 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:46.126133    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Jul 28 18:23:46.134232 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:46.134205    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"]
Jul 28 18:23:46.187975 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:46.187900    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7t4c\" (UniqueName: \"kubernetes.io/projected/7f878dc9-a617-4a5b-80d1-3b9271fd684e-kube-api-access-k7t4c\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk\" (UID: \"7f878dc9-a617-4a5b-80d1-3b9271fd684e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"
Jul 28 18:23:46.187975 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:46.187978    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f878dc9-a617-4a5b-80d1-3b9271fd684e-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk\" (UID: \"7f878dc9-a617-4a5b-80d1-3b9271fd684e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"
Jul 28 18:23:46.188226 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:46.188013    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f878dc9-a617-4a5b-80d1-3b9271fd684e-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk\" (UID: \"7f878dc9-a617-4a5b-80d1-3b9271fd684e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"
Jul 28 18:23:46.188226 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:46.188053    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f878dc9-a617-4a5b-80d1-3b9271fd684e-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk\" (UID: \"7f878dc9-a617-4a5b-80d1-3b9271fd684e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"
Jul 28 18:23:46.188226 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:46.188081    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f878dc9-a617-4a5b-80d1-3b9271fd684e-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk\" (UID: \"7f878dc9-a617-4a5b-80d1-3b9271fd684e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"
Jul 28 18:23:46.288723 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:46.288685    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7t4c\" (UniqueName: \"kubernetes.io/projected/7f878dc9-a617-4a5b-80d1-3b9271fd684e-kube-api-access-k7t4c\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk\" (UID: \"7f878dc9-a617-4a5b-80d1-3b9271fd684e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"
Jul 28 18:23:46.288723 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:46.288726    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f878dc9-a617-4a5b-80d1-3b9271fd684e-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk\" (UID: \"7f878dc9-a617-4a5b-80d1-3b9271fd684e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"
Jul 28 18:23:46.289017 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:46.288758    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f878dc9-a617-4a5b-80d1-3b9271fd684e-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk\" (UID: \"7f878dc9-a617-4a5b-80d1-3b9271fd684e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"
Jul 28 18:23:46.289017 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:46.288798    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f878dc9-a617-4a5b-80d1-3b9271fd684e-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk\" (UID: \"7f878dc9-a617-4a5b-80d1-3b9271fd684e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"
Jul 28 18:23:46.289017 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:46.288822    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f878dc9-a617-4a5b-80d1-3b9271fd684e-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk\" (UID: \"7f878dc9-a617-4a5b-80d1-3b9271fd684e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"
Jul 28 18:23:46.289245 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:46.289223    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f878dc9-a617-4a5b-80d1-3b9271fd684e-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk\" (UID: \"7f878dc9-a617-4a5b-80d1-3b9271fd684e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"
Jul 28 18:23:46.289566 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:46.289545    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f878dc9-a617-4a5b-80d1-3b9271fd684e-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk\" (UID: \"7f878dc9-a617-4a5b-80d1-3b9271fd684e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"
Jul 28 18:23:46.289566 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:46.289558    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f878dc9-a617-4a5b-80d1-3b9271fd684e-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk\" (UID: \"7f878dc9-a617-4a5b-80d1-3b9271fd684e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"
Jul 28 18:23:46.291222 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:46.291200    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f878dc9-a617-4a5b-80d1-3b9271fd684e-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk\" (UID: \"7f878dc9-a617-4a5b-80d1-3b9271fd684e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"
Jul 28 18:23:46.298326 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:46.298302    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7t4c\" (UniqueName: \"kubernetes.io/projected/7f878dc9-a617-4a5b-80d1-3b9271fd684e-kube-api-access-k7t4c\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk\" (UID: \"7f878dc9-a617-4a5b-80d1-3b9271fd684e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"
Jul 28 18:23:46.324001 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:46.323960    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:23:46.349588 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:46.349500    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc" podUID="3636a612-4b85-4aef-9143-39a4de422984" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.60:8643/healthz\": dial tcp 10.133.0.60:8643: connect: connection refused"
Jul 28 18:23:46.433368 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:46.433330    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"
Jul 28 18:23:46.570307 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:46.566384    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"]
Jul 28 18:23:46.570307 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:23:46.570179    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f878dc9_a617_4a5b_80d1_3b9271fd684e.slice/crio-efd5dccc71e23529ea2393720f8be55b039ff9c3d3aafed67a679d640eaadf26 WatchSource:0}: Error finding container efd5dccc71e23529ea2393720f8be55b039ff9c3d3aafed67a679d640eaadf26: Status 404 returned error can't find the container with id efd5dccc71e23529ea2393720f8be55b039ff9c3d3aafed67a679d640eaadf26
Jul 28 18:23:47.478179 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:47.478142    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk" event={"ID":"7f878dc9-a617-4a5b-80d1-3b9271fd684e","Type":"ContainerStarted","Data":"df6db79cf642630b6b6ffe4170c82313f5bf4be602e6357d5de860ae36276b1a"}
Jul 28 18:23:47.478591 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:47.478200    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk" event={"ID":"7f878dc9-a617-4a5b-80d1-3b9271fd684e","Type":"ContainerStarted","Data":"efd5dccc71e23529ea2393720f8be55b039ff9c3d3aafed67a679d640eaadf26"}
Jul 28 18:23:51.349444 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:51.349396    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc" podUID="3636a612-4b85-4aef-9143-39a4de422984" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.60:8643/healthz\": dial tcp 10.133.0.60:8643: connect: connection refused"
Jul 28 18:23:51.349865 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:51.349527    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"
Jul 28 18:23:52.496161 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:52.496115    2581 generic.go:356] "Generic (PLEG): container finished" podID="7f878dc9-a617-4a5b-80d1-3b9271fd684e" containerID="df6db79cf642630b6b6ffe4170c82313f5bf4be602e6357d5de860ae36276b1a" exitCode=0
Jul 28 18:23:52.496161 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:52.496156    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk" event={"ID":"7f878dc9-a617-4a5b-80d1-3b9271fd684e","Type":"ContainerDied","Data":"df6db79cf642630b6b6ffe4170c82313f5bf4be602e6357d5de860ae36276b1a"}
Jul 28 18:23:53.501393 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:53.501354    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk" event={"ID":"7f878dc9-a617-4a5b-80d1-3b9271fd684e","Type":"ContainerStarted","Data":"5e41358c3274dbf625443748e35a2bc83daebbe14fc655708ea87861b39da828"}
Jul 28 18:23:53.501813 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:53.501423    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk" event={"ID":"7f878dc9-a617-4a5b-80d1-3b9271fd684e","Type":"ContainerStarted","Data":"5e22cf523e9bbac678d2a4da39576b847e02b5b5fd508ad9198a48ef1b28c82a"}
Jul 28 18:23:53.501813 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:53.501627    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"
Jul 28 18:23:53.524636 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:53.524576    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk" podStartSLOduration=2.621348195 podStartE2EDuration="7.52455882s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.903210625s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:23:53.523157733 +0000 UTC m=+4481.797774166" watchObservedRunningTime="2026-07-28 18:23:53.52455882 +0000 UTC m=+4481.799175251"
Jul 28 18:23:54.505104 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:54.505073    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"
Jul 28 18:23:54.506502 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:54.506469    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk" podUID="7f878dc9-a617-4a5b-80d1-3b9271fd684e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Jul 28 18:23:55.509026 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:55.508985    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk" podUID="7f878dc9-a617-4a5b-80d1-3b9271fd684e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Jul 28 18:23:56.349860 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:23:56.349809    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc" podUID="3636a612-4b85-4aef-9143-39a4de422984" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.60:8643/healthz\": dial tcp 10.133.0.60:8643: connect: connection refused"
Jul 28 18:24:00.514246 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:00.514100    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"
Jul 28 18:24:00.514733 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:00.514695    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk" podUID="7f878dc9-a617-4a5b-80d1-3b9271fd684e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Jul 28 18:24:01.349918 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:01.349863    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc" podUID="3636a612-4b85-4aef-9143-39a4de422984" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.60:8643/healthz\": dial tcp 10.133.0.60:8643: connect: connection refused"
Jul 28 18:24:06.349540 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:06.349437    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc" podUID="3636a612-4b85-4aef-9143-39a4de422984" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.60:8643/healthz\": dial tcp 10.133.0.60:8643: connect: connection refused"
Jul 28 18:24:06.907462 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:06.907413    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:24:10.515528 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:10.515489    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"
Jul 28 18:24:11.349678 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:11.349580    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc" podUID="3636a612-4b85-4aef-9143-39a4de422984" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.60:8643/healthz\": dial tcp 10.133.0.60:8643: connect: connection refused"
Jul 28 18:24:11.568722 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:11.568689    2581 generic.go:356] "Generic (PLEG): container finished" podID="3636a612-4b85-4aef-9143-39a4de422984" containerID="060c262bf7c156d0090b5de69fc06e6c217d2bd6b7f911fb12cb6379485c3953" exitCode=137
Jul 28 18:24:11.569060 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:11.568764    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc" event={"ID":"3636a612-4b85-4aef-9143-39a4de422984","Type":"ContainerDied","Data":"060c262bf7c156d0090b5de69fc06e6c217d2bd6b7f911fb12cb6379485c3953"}
Jul 28 18:24:11.569060 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:11.568823    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc" event={"ID":"3636a612-4b85-4aef-9143-39a4de422984","Type":"ContainerDied","Data":"9c1ad8e1a884e54dad7036b2fa523398224fcded09363df942f7ed84153de690"}
Jul 28 18:24:11.569060 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:11.568844    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c1ad8e1a884e54dad7036b2fa523398224fcded09363df942f7ed84153de690"
Jul 28 18:24:11.571042 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:11.571022    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"
Jul 28 18:24:11.681997 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:11.681960    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3636a612-4b85-4aef-9143-39a4de422984-kube-rbac-proxy-sar-config\") pod \"3636a612-4b85-4aef-9143-39a4de422984\" (UID: \"3636a612-4b85-4aef-9143-39a4de422984\") "
Jul 28 18:24:11.682167 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:11.682001    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3636a612-4b85-4aef-9143-39a4de422984-proxy-tls\") pod \"3636a612-4b85-4aef-9143-39a4de422984\" (UID: \"3636a612-4b85-4aef-9143-39a4de422984\") "
Jul 28 18:24:11.682167 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:11.682026    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpdgt\" (UniqueName: \"kubernetes.io/projected/3636a612-4b85-4aef-9143-39a4de422984-kube-api-access-lpdgt\") pod \"3636a612-4b85-4aef-9143-39a4de422984\" (UID: \"3636a612-4b85-4aef-9143-39a4de422984\") "
Jul 28 18:24:11.682167 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:11.682092    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3636a612-4b85-4aef-9143-39a4de422984-cabundle-cert\") pod \"3636a612-4b85-4aef-9143-39a4de422984\" (UID: \"3636a612-4b85-4aef-9143-39a4de422984\") "
Jul 28 18:24:11.682167 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:11.682134    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3636a612-4b85-4aef-9143-39a4de422984-kserve-provision-location\") pod \"3636a612-4b85-4aef-9143-39a4de422984\" (UID: \"3636a612-4b85-4aef-9143-39a4de422984\") "
Jul 28 18:24:11.682390 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:11.682364    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3636a612-4b85-4aef-9143-39a4de422984-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3636a612-4b85-4aef-9143-39a4de422984" (UID: "3636a612-4b85-4aef-9143-39a4de422984"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:24:11.682498 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:11.682473    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3636a612-4b85-4aef-9143-39a4de422984-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3636a612-4b85-4aef-9143-39a4de422984" (UID: "3636a612-4b85-4aef-9143-39a4de422984"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:24:11.684207 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:11.684177    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3636a612-4b85-4aef-9143-39a4de422984-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3636a612-4b85-4aef-9143-39a4de422984" (UID: "3636a612-4b85-4aef-9143-39a4de422984"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:24:11.684327 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:11.684270    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3636a612-4b85-4aef-9143-39a4de422984-kube-api-access-lpdgt" (OuterVolumeSpecName: "kube-api-access-lpdgt") pod "3636a612-4b85-4aef-9143-39a4de422984" (UID: "3636a612-4b85-4aef-9143-39a4de422984"). InnerVolumeSpecName "kube-api-access-lpdgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:24:11.693498 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:11.693466    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3636a612-4b85-4aef-9143-39a4de422984-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3636a612-4b85-4aef-9143-39a4de422984" (UID: "3636a612-4b85-4aef-9143-39a4de422984"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:24:11.782902 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:11.782867    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3636a612-4b85-4aef-9143-39a4de422984-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:24:11.782902 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:11.782897    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3636a612-4b85-4aef-9143-39a4de422984-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:24:11.782902 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:11.782906    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3636a612-4b85-4aef-9143-39a4de422984-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:24:11.783174 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:11.782918    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3636a612-4b85-4aef-9143-39a4de422984-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:24:11.783174 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:11.782927    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpdgt\" (UniqueName: \"kubernetes.io/projected/3636a612-4b85-4aef-9143-39a4de422984-kube-api-access-lpdgt\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:24:12.463161 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:12.463129    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 18:24:12.469212 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:12.469190    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 18:24:12.571723 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:12.571640    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"
Jul 28 18:24:12.590007 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:12.589969    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"]
Jul 28 18:24:12.593780 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:12.593751    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-km2gc"]
Jul 28 18:24:14.290318 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:14.290279    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3636a612-4b85-4aef-9143-39a4de422984" path="/var/lib/kubelet/pods/3636a612-4b85-4aef-9143-39a4de422984/volumes"
Jul 28 18:24:27.431070 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:27.431030    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"]
Jul 28 18:24:27.431667 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:27.431290    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk" podUID="7f878dc9-a617-4a5b-80d1-3b9271fd684e" containerName="kserve-container" containerID="cri-o://5e22cf523e9bbac678d2a4da39576b847e02b5b5fd508ad9198a48ef1b28c82a" gracePeriod=30
Jul 28 18:24:27.431667 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:27.431351    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk" podUID="7f878dc9-a617-4a5b-80d1-3b9271fd684e" containerName="kube-rbac-proxy" containerID="cri-o://5e41358c3274dbf625443748e35a2bc83daebbe14fc655708ea87861b39da828" gracePeriod=30
Jul 28 18:24:27.466447 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:27.466413    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:24:27.627020 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:27.626985    2581 generic.go:356] "Generic (PLEG): container finished" podID="7f878dc9-a617-4a5b-80d1-3b9271fd684e" containerID="5e41358c3274dbf625443748e35a2bc83daebbe14fc655708ea87861b39da828" exitCode=2
Jul 28 18:24:27.627199 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:27.627044    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk" event={"ID":"7f878dc9-a617-4a5b-80d1-3b9271fd684e","Type":"ContainerDied","Data":"5e41358c3274dbf625443748e35a2bc83daebbe14fc655708ea87861b39da828"}
Jul 28 18:24:29.839492 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:29.839454    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:24:30.509459 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:30.509409    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk" podUID="7f878dc9-a617-4a5b-80d1-3b9271fd684e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.61:8643/healthz\": dial tcp 10.133.0.61:8643: connect: connection refused"
Jul 28 18:24:32.572341 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.572299    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:24:32.605269 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.605232    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"]
Jul 28 18:24:32.605573 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.605557    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3636a612-4b85-4aef-9143-39a4de422984" containerName="kserve-container"
Jul 28 18:24:32.605655 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.605577    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3636a612-4b85-4aef-9143-39a4de422984" containerName="kserve-container"
Jul 28 18:24:32.605655 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.605590    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3636a612-4b85-4aef-9143-39a4de422984" containerName="kube-rbac-proxy"
Jul 28 18:24:32.605655 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.605598    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3636a612-4b85-4aef-9143-39a4de422984" containerName="kube-rbac-proxy"
Jul 28 18:24:32.605655 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.605627    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3636a612-4b85-4aef-9143-39a4de422984" containerName="storage-initializer"
Jul 28 18:24:32.605655 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.605637    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3636a612-4b85-4aef-9143-39a4de422984" containerName="storage-initializer"
Jul 28 18:24:32.605924 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.605708    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3636a612-4b85-4aef-9143-39a4de422984" containerName="kserve-container"
Jul 28 18:24:32.605924 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.605724    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3636a612-4b85-4aef-9143-39a4de422984" containerName="kube-rbac-proxy"
Jul 28 18:24:32.608903 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.608879    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"
Jul 28 18:24:32.611462 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.611445    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Jul 28 18:24:32.611564 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.611448    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Jul 28 18:24:32.621253 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.621230    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"]
Jul 28 18:24:32.743688 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.743650    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57f857de-f216-43a2-9ec7-146dff416dd8-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-vvf7p\" (UID: \"57f857de-f216-43a2-9ec7-146dff416dd8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"
Jul 28 18:24:32.743688 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.743695    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57f857de-f216-43a2-9ec7-146dff416dd8-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-vvf7p\" (UID: \"57f857de-f216-43a2-9ec7-146dff416dd8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"
Jul 28 18:24:32.743985 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.743733    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57f857de-f216-43a2-9ec7-146dff416dd8-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-vvf7p\" (UID: \"57f857de-f216-43a2-9ec7-146dff416dd8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"
Jul 28 18:24:32.743985 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.743787    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57f857de-f216-43a2-9ec7-146dff416dd8-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-vvf7p\" (UID: \"57f857de-f216-43a2-9ec7-146dff416dd8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"
Jul 28 18:24:32.743985 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.743810    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfwmk\" (UniqueName: \"kubernetes.io/projected/57f857de-f216-43a2-9ec7-146dff416dd8-kube-api-access-vfwmk\") pod \"isvc-triton-predictor-5d7db58676-vvf7p\" (UID: \"57f857de-f216-43a2-9ec7-146dff416dd8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"
Jul 28 18:24:32.842926 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.842827    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:24:32.844833 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.844811    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57f857de-f216-43a2-9ec7-146dff416dd8-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-vvf7p\" (UID: \"57f857de-f216-43a2-9ec7-146dff416dd8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"
Jul 28 18:24:32.844913 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.844846    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57f857de-f216-43a2-9ec7-146dff416dd8-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-vvf7p\" (UID: \"57f857de-f216-43a2-9ec7-146dff416dd8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"
Jul 28 18:24:32.844913 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.844881    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57f857de-f216-43a2-9ec7-146dff416dd8-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-vvf7p\" (UID: \"57f857de-f216-43a2-9ec7-146dff416dd8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"
Jul 28 18:24:32.845039 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.844993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57f857de-f216-43a2-9ec7-146dff416dd8-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-vvf7p\" (UID: \"57f857de-f216-43a2-9ec7-146dff416dd8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"
Jul 28 18:24:32.845089 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.845046    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfwmk\" (UniqueName: \"kubernetes.io/projected/57f857de-f216-43a2-9ec7-146dff416dd8-kube-api-access-vfwmk\") pod \"isvc-triton-predictor-5d7db58676-vvf7p\" (UID: \"57f857de-f216-43a2-9ec7-146dff416dd8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"
Jul 28 18:24:32.845392 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.845361    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57f857de-f216-43a2-9ec7-146dff416dd8-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-vvf7p\" (UID: \"57f857de-f216-43a2-9ec7-146dff416dd8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"
Jul 28 18:24:32.845602 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.845582    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57f857de-f216-43a2-9ec7-146dff416dd8-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-vvf7p\" (UID: \"57f857de-f216-43a2-9ec7-146dff416dd8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"
Jul 28 18:24:32.845680 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.845584    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57f857de-f216-43a2-9ec7-146dff416dd8-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-vvf7p\" (UID: \"57f857de-f216-43a2-9ec7-146dff416dd8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"
Jul 28 18:24:32.847369 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.847348    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57f857de-f216-43a2-9ec7-146dff416dd8-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-vvf7p\" (UID: \"57f857de-f216-43a2-9ec7-146dff416dd8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"
Jul 28 18:24:32.853729 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.853705    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfwmk\" (UniqueName: \"kubernetes.io/projected/57f857de-f216-43a2-9ec7-146dff416dd8-kube-api-access-vfwmk\") pod \"isvc-triton-predictor-5d7db58676-vvf7p\" (UID: \"57f857de-f216-43a2-9ec7-146dff416dd8\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"
Jul 28 18:24:32.921753 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:32.921716    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"
Jul 28 18:24:33.051379 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:33.051343    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"]
Jul 28 18:24:33.054618 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:24:33.054590    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f857de_f216_43a2_9ec7_146dff416dd8.slice/crio-faac5fb18cabb4d560240f212268febfb351c418033eaf68c6151f0f73176129 WatchSource:0}: Error finding container faac5fb18cabb4d560240f212268febfb351c418033eaf68c6151f0f73176129: Status 404 returned error can't find the container with id faac5fb18cabb4d560240f212268febfb351c418033eaf68c6151f0f73176129
Jul 28 18:24:33.650910 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:33.650870    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p" event={"ID":"57f857de-f216-43a2-9ec7-146dff416dd8","Type":"ContainerStarted","Data":"e672b74adac7d2d8da78f15556424a44ddd7c28bb5290d60636d304dc683da1c"}
Jul 28 18:24:33.651322 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:33.650963    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p" event={"ID":"57f857de-f216-43a2-9ec7-146dff416dd8","Type":"ContainerStarted","Data":"faac5fb18cabb4d560240f212268febfb351c418033eaf68c6151f0f73176129"}
Jul 28 18:24:35.509503 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:35.509455    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk" podUID="7f878dc9-a617-4a5b-80d1-3b9271fd684e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.61:8643/healthz\": dial tcp 10.133.0.61:8643: connect: connection refused"
Jul 28 18:24:37.665444 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:37.665407    2581 generic.go:356] "Generic (PLEG): container finished" podID="57f857de-f216-43a2-9ec7-146dff416dd8" containerID="e672b74adac7d2d8da78f15556424a44ddd7c28bb5290d60636d304dc683da1c" exitCode=0
Jul 28 18:24:37.665859 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:37.665476    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p" event={"ID":"57f857de-f216-43a2-9ec7-146dff416dd8","Type":"ContainerDied","Data":"e672b74adac7d2d8da78f15556424a44ddd7c28bb5290d60636d304dc683da1c"}
Jul 28 18:24:40.510354 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:40.510302    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk" podUID="7f878dc9-a617-4a5b-80d1-3b9271fd684e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.61:8643/healthz\": dial tcp 10.133.0.61:8643: connect: connection refused"
Jul 28 18:24:40.510867 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:40.510465    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"
Jul 28 18:24:45.509325 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:45.509277    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk" podUID="7f878dc9-a617-4a5b-80d1-3b9271fd684e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.61:8643/healthz\": dial tcp 10.133.0.61:8643: connect: connection refused"
Jul 28 18:24:50.510286 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:50.510232    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk" podUID="7f878dc9-a617-4a5b-80d1-3b9271fd684e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.61:8643/healthz\": dial tcp 10.133.0.61:8643: connect: connection refused"
Jul 28 18:24:55.509385 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:55.509329    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk" podUID="7f878dc9-a617-4a5b-80d1-3b9271fd684e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.61:8643/healthz\": dial tcp 10.133.0.61:8643: connect: connection refused"
Jul 28 18:24:57.789766 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:57.789678    2581 generic.go:356] "Generic (PLEG): container finished" podID="7f878dc9-a617-4a5b-80d1-3b9271fd684e" containerID="5e22cf523e9bbac678d2a4da39576b847e02b5b5fd508ad9198a48ef1b28c82a" exitCode=137
Jul 28 18:24:57.790229 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:57.789764    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk" event={"ID":"7f878dc9-a617-4a5b-80d1-3b9271fd684e","Type":"ContainerDied","Data":"5e22cf523e9bbac678d2a4da39576b847e02b5b5fd508ad9198a48ef1b28c82a"}
Jul 28 18:24:58.146813 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:58.146737    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"
Jul 28 18:24:58.261529 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:58.261023    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f878dc9-a617-4a5b-80d1-3b9271fd684e-proxy-tls\") pod \"7f878dc9-a617-4a5b-80d1-3b9271fd684e\" (UID: \"7f878dc9-a617-4a5b-80d1-3b9271fd684e\") "
Jul 28 18:24:58.261529 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:58.261069    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f878dc9-a617-4a5b-80d1-3b9271fd684e-kserve-provision-location\") pod \"7f878dc9-a617-4a5b-80d1-3b9271fd684e\" (UID: \"7f878dc9-a617-4a5b-80d1-3b9271fd684e\") "
Jul 28 18:24:58.261529 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:58.261099    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7t4c\" (UniqueName: \"kubernetes.io/projected/7f878dc9-a617-4a5b-80d1-3b9271fd684e-kube-api-access-k7t4c\") pod \"7f878dc9-a617-4a5b-80d1-3b9271fd684e\" (UID: \"7f878dc9-a617-4a5b-80d1-3b9271fd684e\") "
Jul 28 18:24:58.261529 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:58.261148    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f878dc9-a617-4a5b-80d1-3b9271fd684e-cabundle-cert\") pod \"7f878dc9-a617-4a5b-80d1-3b9271fd684e\" (UID: \"7f878dc9-a617-4a5b-80d1-3b9271fd684e\") "
Jul 28 18:24:58.261529 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:58.261198    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f878dc9-a617-4a5b-80d1-3b9271fd684e-kube-rbac-proxy-sar-config\") pod \"7f878dc9-a617-4a5b-80d1-3b9271fd684e\" (UID: \"7f878dc9-a617-4a5b-80d1-3b9271fd684e\") "
Jul 28 18:24:58.262021 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:58.261883    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f878dc9-a617-4a5b-80d1-3b9271fd684e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7f878dc9-a617-4a5b-80d1-3b9271fd684e" (UID: "7f878dc9-a617-4a5b-80d1-3b9271fd684e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:24:58.262257 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:58.262217    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f878dc9-a617-4a5b-80d1-3b9271fd684e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7f878dc9-a617-4a5b-80d1-3b9271fd684e" (UID: "7f878dc9-a617-4a5b-80d1-3b9271fd684e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:24:58.265911 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:58.265882    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f878dc9-a617-4a5b-80d1-3b9271fd684e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7f878dc9-a617-4a5b-80d1-3b9271fd684e" (UID: "7f878dc9-a617-4a5b-80d1-3b9271fd684e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:24:58.266760 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:58.266727    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f878dc9-a617-4a5b-80d1-3b9271fd684e-kube-api-access-k7t4c" (OuterVolumeSpecName: "kube-api-access-k7t4c") pod "7f878dc9-a617-4a5b-80d1-3b9271fd684e" (UID: "7f878dc9-a617-4a5b-80d1-3b9271fd684e"). InnerVolumeSpecName "kube-api-access-k7t4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:24:58.266874 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:58.266793    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f878dc9-a617-4a5b-80d1-3b9271fd684e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7f878dc9-a617-4a5b-80d1-3b9271fd684e" (UID: "7f878dc9-a617-4a5b-80d1-3b9271fd684e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:24:58.362116 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:58.361921    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f878dc9-a617-4a5b-80d1-3b9271fd684e-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:24:58.362116 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:58.361970    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f878dc9-a617-4a5b-80d1-3b9271fd684e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:24:58.362116 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:58.362010    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f878dc9-a617-4a5b-80d1-3b9271fd684e-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:24:58.362116 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:58.362023    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f878dc9-a617-4a5b-80d1-3b9271fd684e-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:24:58.362116 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:58.362037    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7t4c\" (UniqueName: \"kubernetes.io/projected/7f878dc9-a617-4a5b-80d1-3b9271fd684e-kube-api-access-k7t4c\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:24:58.800535 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:58.800488    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk" event={"ID":"7f878dc9-a617-4a5b-80d1-3b9271fd684e","Type":"ContainerDied","Data":"efd5dccc71e23529ea2393720f8be55b039ff9c3d3aafed67a679d640eaadf26"}
Jul 28 18:24:58.800535 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:58.800530    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"
Jul 28 18:24:58.801366 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:58.800611    2581 scope.go:117] "RemoveContainer" containerID="5e41358c3274dbf625443748e35a2bc83daebbe14fc655708ea87861b39da828"
Jul 28 18:24:58.812602 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:58.812576    2581 scope.go:117] "RemoveContainer" containerID="5e22cf523e9bbac678d2a4da39576b847e02b5b5fd508ad9198a48ef1b28c82a"
Jul 28 18:24:58.823159 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:58.823051    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"]
Jul 28 18:24:58.823916 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:58.823827    2581 scope.go:117] "RemoveContainer" containerID="df6db79cf642630b6b6ffe4170c82313f5bf4be602e6357d5de860ae36276b1a"
Jul 28 18:24:58.827588 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:24:58.827559    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zzkmk"]
Jul 28 18:25:00.292205 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:25:00.292171    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f878dc9-a617-4a5b-80d1-3b9271fd684e" path="/var/lib/kubelet/pods/7f878dc9-a617-4a5b-80d1-3b9271fd684e/volumes"
Jul 28 18:26:33.202803 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:33.202762    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p" event={"ID":"57f857de-f216-43a2-9ec7-146dff416dd8","Type":"ContainerStarted","Data":"48a40a147a0adc0c260589b591eb15551735c110105174637a95a8fcaee2057d"}
Jul 28 18:26:33.203361 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:33.202829    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p" event={"ID":"57f857de-f216-43a2-9ec7-146dff416dd8","Type":"ContainerStarted","Data":"402f4be93d1bb3b8dd1f96c46eee491f1410a746fa51093dbf676ac70124057f"}
Jul 28 18:26:33.203361 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:33.202904    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"
Jul 28 18:26:33.227493 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:33.227437    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p" podStartSLOduration=2.408274161 podStartE2EDuration="2m1.227417713s" totalImagesPullingTime="1m54.88063716s" totalInitContainerRuntime="3.938506392s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:26:33.227084734 +0000 UTC m=+4641.501701163" watchObservedRunningTime="2026-07-28 18:26:33.227417713 +0000 UTC m=+4641.502034147"
Jul 28 18:26:34.207149 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:34.207105    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"
Jul 28 18:26:34.208545 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:34.208512    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p" podUID="57f857de-f216-43a2-9ec7-146dff416dd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 28 18:26:35.210639 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:35.210602    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p" podUID="57f857de-f216-43a2-9ec7-146dff416dd8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 28 18:26:40.214934 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:40.214903    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"
Jul 28 18:26:40.216824 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:40.216796    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"
Jul 28 18:26:44.250414 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:44.250374    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"]
Jul 28 18:26:44.250893 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:44.250630    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p" podUID="57f857de-f216-43a2-9ec7-146dff416dd8" containerName="kserve-container" containerID="cri-o://402f4be93d1bb3b8dd1f96c46eee491f1410a746fa51093dbf676ac70124057f" gracePeriod=30
Jul 28 18:26:44.250893 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:44.250714    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p" podUID="57f857de-f216-43a2-9ec7-146dff416dd8" containerName="kube-rbac-proxy" containerID="cri-o://48a40a147a0adc0c260589b591eb15551735c110105174637a95a8fcaee2057d" gracePeriod=30
Jul 28 18:26:44.308134 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:44.308094    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:26:45.211733 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:45.211691    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p" podUID="57f857de-f216-43a2-9ec7-146dff416dd8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.62:8643/healthz\": dial tcp 10.133.0.62:8643: connect: connection refused"
Jul 28 18:26:45.248535 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:45.248500    2581 generic.go:356] "Generic (PLEG): container finished" podID="57f857de-f216-43a2-9ec7-146dff416dd8" containerID="48a40a147a0adc0c260589b591eb15551735c110105174637a95a8fcaee2057d" exitCode=2
Jul 28 18:26:45.248713 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:45.248569    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p" event={"ID":"57f857de-f216-43a2-9ec7-146dff416dd8","Type":"ContainerDied","Data":"48a40a147a0adc0c260589b591eb15551735c110105174637a95a8fcaee2057d"}
Jul 28 18:26:47.257768 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:47.257735    2581 generic.go:356] "Generic (PLEG): container finished" podID="57f857de-f216-43a2-9ec7-146dff416dd8" containerID="402f4be93d1bb3b8dd1f96c46eee491f1410a746fa51093dbf676ac70124057f" exitCode=0
Jul 28 18:26:47.258175 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:47.257807    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p" event={"ID":"57f857de-f216-43a2-9ec7-146dff416dd8","Type":"ContainerDied","Data":"402f4be93d1bb3b8dd1f96c46eee491f1410a746fa51093dbf676ac70124057f"}
Jul 28 18:26:47.320808 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:47.320722    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"
Jul 28 18:26:47.454248 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:47.454210    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfwmk\" (UniqueName: \"kubernetes.io/projected/57f857de-f216-43a2-9ec7-146dff416dd8-kube-api-access-vfwmk\") pod \"57f857de-f216-43a2-9ec7-146dff416dd8\" (UID: \"57f857de-f216-43a2-9ec7-146dff416dd8\") "
Jul 28 18:26:47.454450 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:47.454276    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57f857de-f216-43a2-9ec7-146dff416dd8-kube-rbac-proxy-sar-config\") pod \"57f857de-f216-43a2-9ec7-146dff416dd8\" (UID: \"57f857de-f216-43a2-9ec7-146dff416dd8\") "
Jul 28 18:26:47.454450 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:47.454379    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57f857de-f216-43a2-9ec7-146dff416dd8-cabundle-cert\") pod \"57f857de-f216-43a2-9ec7-146dff416dd8\" (UID: \"57f857de-f216-43a2-9ec7-146dff416dd8\") "
Jul 28 18:26:47.454450 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:47.454428    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57f857de-f216-43a2-9ec7-146dff416dd8-proxy-tls\") pod \"57f857de-f216-43a2-9ec7-146dff416dd8\" (UID: \"57f857de-f216-43a2-9ec7-146dff416dd8\") "
Jul 28 18:26:47.454632 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:47.454468    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57f857de-f216-43a2-9ec7-146dff416dd8-kserve-provision-location\") pod \"57f857de-f216-43a2-9ec7-146dff416dd8\" (UID: \"57f857de-f216-43a2-9ec7-146dff416dd8\") "
Jul 28 18:26:47.455238 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:47.455204    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57f857de-f216-43a2-9ec7-146dff416dd8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "57f857de-f216-43a2-9ec7-146dff416dd8" (UID: "57f857de-f216-43a2-9ec7-146dff416dd8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:26:47.455238 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:47.455214    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57f857de-f216-43a2-9ec7-146dff416dd8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "57f857de-f216-43a2-9ec7-146dff416dd8" (UID: "57f857de-f216-43a2-9ec7-146dff416dd8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:26:47.456056 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:47.456032    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57f857de-f216-43a2-9ec7-146dff416dd8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:26:47.456165 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:47.456059    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57f857de-f216-43a2-9ec7-146dff416dd8-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:26:47.456629 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:47.456354    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f857de-f216-43a2-9ec7-146dff416dd8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "57f857de-f216-43a2-9ec7-146dff416dd8" (UID: "57f857de-f216-43a2-9ec7-146dff416dd8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:26:47.461199 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:47.456812    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f857de-f216-43a2-9ec7-146dff416dd8-kube-api-access-vfwmk" (OuterVolumeSpecName: "kube-api-access-vfwmk") pod "57f857de-f216-43a2-9ec7-146dff416dd8" (UID: "57f857de-f216-43a2-9ec7-146dff416dd8"). InnerVolumeSpecName "kube-api-access-vfwmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:26:47.461603 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:47.461559    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57f857de-f216-43a2-9ec7-146dff416dd8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "57f857de-f216-43a2-9ec7-146dff416dd8" (UID: "57f857de-f216-43a2-9ec7-146dff416dd8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:26:47.557032 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:47.556987    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57f857de-f216-43a2-9ec7-146dff416dd8-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:26:47.557032 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:47.557023    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57f857de-f216-43a2-9ec7-146dff416dd8-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:26:47.557032 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:47.557033    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfwmk\" (UniqueName: \"kubernetes.io/projected/57f857de-f216-43a2-9ec7-146dff416dd8-kube-api-access-vfwmk\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:26:48.263057 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:48.263019    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p" event={"ID":"57f857de-f216-43a2-9ec7-146dff416dd8","Type":"ContainerDied","Data":"faac5fb18cabb4d560240f212268febfb351c418033eaf68c6151f0f73176129"}
Jul 28 18:26:48.263057 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:48.263052    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"
Jul 28 18:26:48.263574 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:48.263109    2581 scope.go:117] "RemoveContainer" containerID="48a40a147a0adc0c260589b591eb15551735c110105174637a95a8fcaee2057d"
Jul 28 18:26:48.271330 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:48.271225    2581 scope.go:117] "RemoveContainer" containerID="402f4be93d1bb3b8dd1f96c46eee491f1410a746fa51093dbf676ac70124057f"
Jul 28 18:26:48.279019 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:48.278997    2581 scope.go:117] "RemoveContainer" containerID="e672b74adac7d2d8da78f15556424a44ddd7c28bb5290d60636d304dc683da1c"
Jul 28 18:26:48.290288 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:48.290268    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"]
Jul 28 18:26:48.290387 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:48.290300    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-vvf7p"]
Jul 28 18:26:49.351538 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.351495    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:26:49.408157 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.408118    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"]
Jul 28 18:26:49.408511 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.408495    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f878dc9-a617-4a5b-80d1-3b9271fd684e" containerName="kserve-container"
Jul 28 18:26:49.408560 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.408515    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f878dc9-a617-4a5b-80d1-3b9271fd684e" containerName="kserve-container"
Jul 28 18:26:49.408560 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.408529    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f878dc9-a617-4a5b-80d1-3b9271fd684e" containerName="kube-rbac-proxy"
Jul 28 18:26:49.408560 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.408539    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f878dc9-a617-4a5b-80d1-3b9271fd684e" containerName="kube-rbac-proxy"
Jul 28 18:26:49.408560 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.408558    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f878dc9-a617-4a5b-80d1-3b9271fd684e" containerName="storage-initializer"
Jul 28 18:26:49.408685 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.408567    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f878dc9-a617-4a5b-80d1-3b9271fd684e" containerName="storage-initializer"
Jul 28 18:26:49.408685 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.408577    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57f857de-f216-43a2-9ec7-146dff416dd8" containerName="kube-rbac-proxy"
Jul 28 18:26:49.408685 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.408585    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="57f857de-f216-43a2-9ec7-146dff416dd8" containerName="kube-rbac-proxy"
Jul 28 18:26:49.408685 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.408595    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57f857de-f216-43a2-9ec7-146dff416dd8" containerName="storage-initializer"
Jul 28 18:26:49.408685 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.408604    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="57f857de-f216-43a2-9ec7-146dff416dd8" containerName="storage-initializer"
Jul 28 18:26:49.408685 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.408613    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57f857de-f216-43a2-9ec7-146dff416dd8" containerName="kserve-container"
Jul 28 18:26:49.408685 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.408621    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="57f857de-f216-43a2-9ec7-146dff416dd8" containerName="kserve-container"
Jul 28 18:26:49.408685 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.408680    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f878dc9-a617-4a5b-80d1-3b9271fd684e" containerName="kserve-container"
Jul 28 18:26:49.408924 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.408689    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="57f857de-f216-43a2-9ec7-146dff416dd8" containerName="kube-rbac-proxy"
Jul 28 18:26:49.408924 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.408699    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f878dc9-a617-4a5b-80d1-3b9271fd684e" containerName="kube-rbac-proxy"
Jul 28 18:26:49.408924 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.408710    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="57f857de-f216-43a2-9ec7-146dff416dd8" containerName="kserve-container"
Jul 28 18:26:49.444555 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.444520    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"]
Jul 28 18:26:49.444700 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.444628    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"
Jul 28 18:26:49.447579 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.447559    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:26:49.447730 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.447681    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:26:49.447730 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.447692    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:26:49.447860 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.447727    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Jul 28 18:26:49.448931 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.448910    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 18:26:49.449049 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.448965    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:26:49.449049 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.449025    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Jul 28 18:26:49.575666 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.575634    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-lr2ht\" (UID: \"4325596a-d3e1-4ad6-9c99-c7d50c2619c2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"
Jul 28 18:26:49.575666 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.575667    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdkdn\" (UniqueName: \"kubernetes.io/projected/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-kube-api-access-fdkdn\") pod \"isvc-xgboost-predictor-7566d65bf7-lr2ht\" (UID: \"4325596a-d3e1-4ad6-9c99-c7d50c2619c2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"
Jul 28 18:26:49.575915 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.575691    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-lr2ht\" (UID: \"4325596a-d3e1-4ad6-9c99-c7d50c2619c2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"
Jul 28 18:26:49.575915 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.575785    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-lr2ht\" (UID: \"4325596a-d3e1-4ad6-9c99-c7d50c2619c2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"
Jul 28 18:26:49.575915 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.575838    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-lr2ht\" (UID: \"4325596a-d3e1-4ad6-9c99-c7d50c2619c2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"
Jul 28 18:26:49.653148 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.653116    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:26:49.677282 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.677246    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-lr2ht\" (UID: \"4325596a-d3e1-4ad6-9c99-c7d50c2619c2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"
Jul 28 18:26:49.677453 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.677321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-lr2ht\" (UID: \"4325596a-d3e1-4ad6-9c99-c7d50c2619c2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"
Jul 28 18:26:49.677453 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.677375    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-lr2ht\" (UID: \"4325596a-d3e1-4ad6-9c99-c7d50c2619c2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"
Jul 28 18:26:49.677453 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.677402    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdkdn\" (UniqueName: \"kubernetes.io/projected/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-kube-api-access-fdkdn\") pod \"isvc-xgboost-predictor-7566d65bf7-lr2ht\" (UID: \"4325596a-d3e1-4ad6-9c99-c7d50c2619c2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"
Jul 28 18:26:49.677453 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.677442    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-lr2ht\" (UID: \"4325596a-d3e1-4ad6-9c99-c7d50c2619c2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"
Jul 28 18:26:49.678002 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.677980    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-lr2ht\" (UID: \"4325596a-d3e1-4ad6-9c99-c7d50c2619c2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"
Jul 28 18:26:49.678133 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.678085    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-lr2ht\" (UID: \"4325596a-d3e1-4ad6-9c99-c7d50c2619c2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"
Jul 28 18:26:49.678194 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.678154    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-lr2ht\" (UID: \"4325596a-d3e1-4ad6-9c99-c7d50c2619c2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"
Jul 28 18:26:49.679858 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.679839    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-lr2ht\" (UID: \"4325596a-d3e1-4ad6-9c99-c7d50c2619c2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"
Jul 28 18:26:49.686523 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.686498    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdkdn\" (UniqueName: \"kubernetes.io/projected/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-kube-api-access-fdkdn\") pod \"isvc-xgboost-predictor-7566d65bf7-lr2ht\" (UID: \"4325596a-d3e1-4ad6-9c99-c7d50c2619c2\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"
Jul 28 18:26:49.755117 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.755075    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"
Jul 28 18:26:49.908041 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:49.907962    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"]
Jul 28 18:26:49.911227 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:26:49.911196    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4325596a_d3e1_4ad6_9c99_c7d50c2619c2.slice/crio-c6c516c932c2ac944554c73bf491d85d36dd89299bc5dc045e5ee368b925d454 WatchSource:0}: Error finding container c6c516c932c2ac944554c73bf491d85d36dd89299bc5dc045e5ee368b925d454: Status 404 returned error can't find the container with id c6c516c932c2ac944554c73bf491d85d36dd89299bc5dc045e5ee368b925d454
Jul 28 18:26:50.274244 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:50.274161    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht" event={"ID":"4325596a-d3e1-4ad6-9c99-c7d50c2619c2","Type":"ContainerStarted","Data":"4817207c3309b06c2eeda17f0396855a3deb0472f388bc6c76620ac8dee0a806"}
Jul 28 18:26:50.274244 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:50.274214    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht" event={"ID":"4325596a-d3e1-4ad6-9c99-c7d50c2619c2","Type":"ContainerStarted","Data":"c6c516c932c2ac944554c73bf491d85d36dd89299bc5dc045e5ee368b925d454"}
Jul 28 18:26:50.291080 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:50.291051    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57f857de-f216-43a2-9ec7-146dff416dd8" path="/var/lib/kubelet/pods/57f857de-f216-43a2-9ec7-146dff416dd8/volumes"
Jul 28 18:26:54.290635 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:54.290543    2581 generic.go:356] "Generic (PLEG): container finished" podID="4325596a-d3e1-4ad6-9c99-c7d50c2619c2" containerID="4817207c3309b06c2eeda17f0396855a3deb0472f388bc6c76620ac8dee0a806" exitCode=0
Jul 28 18:26:54.290635 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:26:54.290608    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht" event={"ID":"4325596a-d3e1-4ad6-9c99-c7d50c2619c2","Type":"ContainerDied","Data":"4817207c3309b06c2eeda17f0396855a3deb0472f388bc6c76620ac8dee0a806"}
Jul 28 18:27:03.682539 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:27:03.682496    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:27:14.372232 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:27:14.372193    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht" event={"ID":"4325596a-d3e1-4ad6-9c99-c7d50c2619c2","Type":"ContainerStarted","Data":"57991641adaf0255b40d208ed9ccfef3b643761db4f76af61cb7f980086634e8"}
Jul 28 18:27:14.372678 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:27:14.372259    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht" event={"ID":"4325596a-d3e1-4ad6-9c99-c7d50c2619c2","Type":"ContainerStarted","Data":"5b9e1131e83bb6bba35c11f7009b560f46808103d19fdf433f7668d38ca6ce6b"}
Jul 28 18:27:14.372678 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:27:14.372550    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"
Jul 28 18:27:15.376000 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:27:15.375964    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"
Jul 28 18:27:15.377260 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:27:15.377231    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht" podUID="4325596a-d3e1-4ad6-9c99-c7d50c2619c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Jul 28 18:27:16.379311 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:27:16.379269    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht" podUID="4325596a-d3e1-4ad6-9c99-c7d50c2619c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Jul 28 18:27:21.384040 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:27:21.384012    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"
Jul 28 18:27:21.384601 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:27:21.384574    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht" podUID="4325596a-d3e1-4ad6-9c99-c7d50c2619c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Jul 28 18:27:21.404794 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:27:21.404736    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht" podStartSLOduration=9.494720897 podStartE2EDuration="32.404720989s" totalImagesPullingTime="18.976401543s" totalInitContainerRuntime="3.933598549s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:27:14.394159767 +0000 UTC m=+4682.668776200" watchObservedRunningTime="2026-07-28 18:27:21.404720989 +0000 UTC m=+4689.679337421"
Jul 28 18:27:31.384916 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:27:31.384876    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht" podUID="4325596a-d3e1-4ad6-9c99-c7d50c2619c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Jul 28 18:27:41.385444 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:27:41.385394    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht" podUID="4325596a-d3e1-4ad6-9c99-c7d50c2619c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Jul 28 18:27:51.385272 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:27:51.385230    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht" podUID="4325596a-d3e1-4ad6-9c99-c7d50c2619c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Jul 28 18:28:01.384622 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:01.384570    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht" podUID="4325596a-d3e1-4ad6-9c99-c7d50c2619c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Jul 28 18:28:11.385521 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:11.385477    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht" podUID="4325596a-d3e1-4ad6-9c99-c7d50c2619c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Jul 28 18:28:21.386027 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:21.385991    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"
Jul 28 18:28:29.564290 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:29.564255    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"]
Jul 28 18:28:29.564887 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:29.564504    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht" podUID="4325596a-d3e1-4ad6-9c99-c7d50c2619c2" containerName="kserve-container" containerID="cri-o://5b9e1131e83bb6bba35c11f7009b560f46808103d19fdf433f7668d38ca6ce6b" gracePeriod=30
Jul 28 18:28:29.564887 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:29.564529    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht" podUID="4325596a-d3e1-4ad6-9c99-c7d50c2619c2" containerName="kube-rbac-proxy" containerID="cri-o://57991641adaf0255b40d208ed9ccfef3b643761db4f76af61cb7f980086634e8" gracePeriod=30
Jul 28 18:28:29.582841 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:29.582808    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:28:30.644048 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:30.644017    2581 generic.go:356] "Generic (PLEG): container finished" podID="4325596a-d3e1-4ad6-9c99-c7d50c2619c2" containerID="57991641adaf0255b40d208ed9ccfef3b643761db4f76af61cb7f980086634e8" exitCode=2
Jul 28 18:28:30.644450 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:30.644064    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht" event={"ID":"4325596a-d3e1-4ad6-9c99-c7d50c2619c2","Type":"ContainerDied","Data":"57991641adaf0255b40d208ed9ccfef3b643761db4f76af61cb7f980086634e8"}
Jul 28 18:28:31.379843 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:31.379795    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht" podUID="4325596a-d3e1-4ad6-9c99-c7d50c2619c2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.63:8643/healthz\": dial tcp 10.133.0.63:8643: connect: connection refused"
Jul 28 18:28:31.385247 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:31.385202    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht" podUID="4325596a-d3e1-4ad6-9c99-c7d50c2619c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.63:8080: connect: connection refused"
Jul 28 18:28:31.619998 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:31.619962    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:28:33.314057 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.314023    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"
Jul 28 18:28:33.361669 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.361632    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-cabundle-cert\") pod \"4325596a-d3e1-4ad6-9c99-c7d50c2619c2\" (UID: \"4325596a-d3e1-4ad6-9c99-c7d50c2619c2\") "
Jul 28 18:28:33.361669 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.361677    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-proxy-tls\") pod \"4325596a-d3e1-4ad6-9c99-c7d50c2619c2\" (UID: \"4325596a-d3e1-4ad6-9c99-c7d50c2619c2\") "
Jul 28 18:28:33.361887 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.361704    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-kube-rbac-proxy-sar-config\") pod \"4325596a-d3e1-4ad6-9c99-c7d50c2619c2\" (UID: \"4325596a-d3e1-4ad6-9c99-c7d50c2619c2\") "
Jul 28 18:28:33.361887 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.361738    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-kserve-provision-location\") pod \"4325596a-d3e1-4ad6-9c99-c7d50c2619c2\" (UID: \"4325596a-d3e1-4ad6-9c99-c7d50c2619c2\") "
Jul 28 18:28:33.361887 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.361806    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdkdn\" (UniqueName: \"kubernetes.io/projected/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-kube-api-access-fdkdn\") pod \"4325596a-d3e1-4ad6-9c99-c7d50c2619c2\" (UID: \"4325596a-d3e1-4ad6-9c99-c7d50c2619c2\") "
Jul 28 18:28:33.362174 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.362149    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4325596a-d3e1-4ad6-9c99-c7d50c2619c2" (UID: "4325596a-d3e1-4ad6-9c99-c7d50c2619c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:28:33.362240 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.362160    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4325596a-d3e1-4ad6-9c99-c7d50c2619c2" (UID: "4325596a-d3e1-4ad6-9c99-c7d50c2619c2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:28:33.362240 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.362151    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4325596a-d3e1-4ad6-9c99-c7d50c2619c2" (UID: "4325596a-d3e1-4ad6-9c99-c7d50c2619c2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:28:33.363997 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.363973    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4325596a-d3e1-4ad6-9c99-c7d50c2619c2" (UID: "4325596a-d3e1-4ad6-9c99-c7d50c2619c2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:28:33.364088 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.364067    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-kube-api-access-fdkdn" (OuterVolumeSpecName: "kube-api-access-fdkdn") pod "4325596a-d3e1-4ad6-9c99-c7d50c2619c2" (UID: "4325596a-d3e1-4ad6-9c99-c7d50c2619c2"). InnerVolumeSpecName "kube-api-access-fdkdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:28:33.463156 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.463067    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:28:33.463156 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.463096    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:28:33.463156 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.463105    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdkdn\" (UniqueName: \"kubernetes.io/projected/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-kube-api-access-fdkdn\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:28:33.463156 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.463114    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:28:33.463156 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.463125    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4325596a-d3e1-4ad6-9c99-c7d50c2619c2-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:28:33.656386 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.656344    2581 generic.go:356] "Generic (PLEG): container finished" podID="4325596a-d3e1-4ad6-9c99-c7d50c2619c2" containerID="5b9e1131e83bb6bba35c11f7009b560f46808103d19fdf433f7668d38ca6ce6b" exitCode=0
Jul 28 18:28:33.656386 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.656393    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht" event={"ID":"4325596a-d3e1-4ad6-9c99-c7d50c2619c2","Type":"ContainerDied","Data":"5b9e1131e83bb6bba35c11f7009b560f46808103d19fdf433f7668d38ca6ce6b"}
Jul 28 18:28:33.656619 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.656407    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"
Jul 28 18:28:33.656619 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.656443    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht" event={"ID":"4325596a-d3e1-4ad6-9c99-c7d50c2619c2","Type":"ContainerDied","Data":"c6c516c932c2ac944554c73bf491d85d36dd89299bc5dc045e5ee368b925d454"}
Jul 28 18:28:33.656619 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.656471    2581 scope.go:117] "RemoveContainer" containerID="57991641adaf0255b40d208ed9ccfef3b643761db4f76af61cb7f980086634e8"
Jul 28 18:28:33.666993 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.666971    2581 scope.go:117] "RemoveContainer" containerID="5b9e1131e83bb6bba35c11f7009b560f46808103d19fdf433f7668d38ca6ce6b"
Jul 28 18:28:33.674245 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.674226    2581 scope.go:117] "RemoveContainer" containerID="4817207c3309b06c2eeda17f0396855a3deb0472f388bc6c76620ac8dee0a806"
Jul 28 18:28:33.680320 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.680296    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"]
Jul 28 18:28:33.681698 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.681682    2581 scope.go:117] "RemoveContainer" containerID="57991641adaf0255b40d208ed9ccfef3b643761db4f76af61cb7f980086634e8"
Jul 28 18:28:33.682052 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:28:33.682027    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57991641adaf0255b40d208ed9ccfef3b643761db4f76af61cb7f980086634e8\": container with ID starting with 57991641adaf0255b40d208ed9ccfef3b643761db4f76af61cb7f980086634e8 not found: ID does not exist" containerID="57991641adaf0255b40d208ed9ccfef3b643761db4f76af61cb7f980086634e8"
Jul 28 18:28:33.682134 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.682062    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57991641adaf0255b40d208ed9ccfef3b643761db4f76af61cb7f980086634e8"} err="failed to get container status \"57991641adaf0255b40d208ed9ccfef3b643761db4f76af61cb7f980086634e8\": rpc error: code = NotFound desc = could not find container \"57991641adaf0255b40d208ed9ccfef3b643761db4f76af61cb7f980086634e8\": container with ID starting with 57991641adaf0255b40d208ed9ccfef3b643761db4f76af61cb7f980086634e8 not found: ID does not exist"
Jul 28 18:28:33.682134 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.682081    2581 scope.go:117] "RemoveContainer" containerID="5b9e1131e83bb6bba35c11f7009b560f46808103d19fdf433f7668d38ca6ce6b"
Jul 28 18:28:33.682516 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:28:33.682490    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b9e1131e83bb6bba35c11f7009b560f46808103d19fdf433f7668d38ca6ce6b\": container with ID starting with 5b9e1131e83bb6bba35c11f7009b560f46808103d19fdf433f7668d38ca6ce6b not found: ID does not exist" containerID="5b9e1131e83bb6bba35c11f7009b560f46808103d19fdf433f7668d38ca6ce6b"
Jul 28 18:28:33.682614 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.682522    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b9e1131e83bb6bba35c11f7009b560f46808103d19fdf433f7668d38ca6ce6b"} err="failed to get container status \"5b9e1131e83bb6bba35c11f7009b560f46808103d19fdf433f7668d38ca6ce6b\": rpc error: code = NotFound desc = could not find container \"5b9e1131e83bb6bba35c11f7009b560f46808103d19fdf433f7668d38ca6ce6b\": container with ID starting with 5b9e1131e83bb6bba35c11f7009b560f46808103d19fdf433f7668d38ca6ce6b not found: ID does not exist"
Jul 28 18:28:33.682614 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.682542    2581 scope.go:117] "RemoveContainer" containerID="4817207c3309b06c2eeda17f0396855a3deb0472f388bc6c76620ac8dee0a806"
Jul 28 18:28:33.683021 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:28:33.683003    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4817207c3309b06c2eeda17f0396855a3deb0472f388bc6c76620ac8dee0a806\": container with ID starting with 4817207c3309b06c2eeda17f0396855a3deb0472f388bc6c76620ac8dee0a806 not found: ID does not exist" containerID="4817207c3309b06c2eeda17f0396855a3deb0472f388bc6c76620ac8dee0a806"
Jul 28 18:28:33.683086 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.683026    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4817207c3309b06c2eeda17f0396855a3deb0472f388bc6c76620ac8dee0a806"} err="failed to get container status \"4817207c3309b06c2eeda17f0396855a3deb0472f388bc6c76620ac8dee0a806\": rpc error: code = NotFound desc = could not find container \"4817207c3309b06c2eeda17f0396855a3deb0472f388bc6c76620ac8dee0a806\": container with ID starting with 4817207c3309b06c2eeda17f0396855a3deb0472f388bc6c76620ac8dee0a806 not found: ID does not exist"
Jul 28 18:28:33.683689 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:33.683670    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-lr2ht"]
Jul 28 18:28:34.289985 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.289928    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4325596a-d3e1-4ad6-9c99-c7d50c2619c2" path="/var/lib/kubelet/pods/4325596a-d3e1-4ad6-9c99-c7d50c2619c2/volumes"
Jul 28 18:28:34.637679 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.637597    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:28:34.691848 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.691813    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"]
Jul 28 18:28:34.692185 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.692169    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4325596a-d3e1-4ad6-9c99-c7d50c2619c2" containerName="kserve-container"
Jul 28 18:28:34.692233 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.692189    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4325596a-d3e1-4ad6-9c99-c7d50c2619c2" containerName="kserve-container"
Jul 28 18:28:34.692233 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.692201    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4325596a-d3e1-4ad6-9c99-c7d50c2619c2" containerName="storage-initializer"
Jul 28 18:28:34.692233 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.692210    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4325596a-d3e1-4ad6-9c99-c7d50c2619c2" containerName="storage-initializer"
Jul 28 18:28:34.692233 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.692223    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4325596a-d3e1-4ad6-9c99-c7d50c2619c2" containerName="kube-rbac-proxy"
Jul 28 18:28:34.692233 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.692231    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4325596a-d3e1-4ad6-9c99-c7d50c2619c2" containerName="kube-rbac-proxy"
Jul 28 18:28:34.692400 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.692301    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4325596a-d3e1-4ad6-9c99-c7d50c2619c2" containerName="kube-rbac-proxy"
Jul 28 18:28:34.692400 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.692310    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4325596a-d3e1-4ad6-9c99-c7d50c2619c2" containerName="kserve-container"
Jul 28 18:28:34.697061 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.697039    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"
Jul 28 18:28:34.700156 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.700135    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:28:34.700364 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.700340    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:28:34.700456 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.700384    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 28 18:28:34.700516 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.700469    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 18:28:34.701396 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.701367    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Jul 28 18:28:34.701396 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.701390    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:28:34.701508 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.701374    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:28:34.705810 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.705788    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"]
Jul 28 18:28:34.775292 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.775247    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbbed506-e8ad-44ab-b312-1777d527702c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc\" (UID: \"fbbed506-e8ad-44ab-b312-1777d527702c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"
Jul 28 18:28:34.775292 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.775293    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbbed506-e8ad-44ab-b312-1777d527702c-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc\" (UID: \"fbbed506-e8ad-44ab-b312-1777d527702c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"
Jul 28 18:28:34.775560 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.775310    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbbed506-e8ad-44ab-b312-1777d527702c-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc\" (UID: \"fbbed506-e8ad-44ab-b312-1777d527702c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"
Jul 28 18:28:34.775560 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.775416    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbbed506-e8ad-44ab-b312-1777d527702c-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc\" (UID: \"fbbed506-e8ad-44ab-b312-1777d527702c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"
Jul 28 18:28:34.775560 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.775460    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zb9n\" (UniqueName: \"kubernetes.io/projected/fbbed506-e8ad-44ab-b312-1777d527702c-kube-api-access-4zb9n\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc\" (UID: \"fbbed506-e8ad-44ab-b312-1777d527702c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"
Jul 28 18:28:34.876004 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.875911    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbbed506-e8ad-44ab-b312-1777d527702c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc\" (UID: \"fbbed506-e8ad-44ab-b312-1777d527702c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"
Jul 28 18:28:34.876004 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.875988    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbbed506-e8ad-44ab-b312-1777d527702c-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc\" (UID: \"fbbed506-e8ad-44ab-b312-1777d527702c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"
Jul 28 18:28:34.876261 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.876015    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbbed506-e8ad-44ab-b312-1777d527702c-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc\" (UID: \"fbbed506-e8ad-44ab-b312-1777d527702c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"
Jul 28 18:28:34.876261 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.876048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbbed506-e8ad-44ab-b312-1777d527702c-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc\" (UID: \"fbbed506-e8ad-44ab-b312-1777d527702c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"
Jul 28 18:28:34.876261 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.876078    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zb9n\" (UniqueName: \"kubernetes.io/projected/fbbed506-e8ad-44ab-b312-1777d527702c-kube-api-access-4zb9n\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc\" (UID: \"fbbed506-e8ad-44ab-b312-1777d527702c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"
Jul 28 18:28:34.876261 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:28:34.876164    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-serving-cert: secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Jul 28 18:28:34.876261 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:28:34.876236    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbbed506-e8ad-44ab-b312-1777d527702c-proxy-tls podName:fbbed506-e8ad-44ab-b312-1777d527702c nodeName:}" failed. No retries permitted until 2026-07-28 18:28:35.376218091 +0000 UTC m=+4763.650834513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fbbed506-e8ad-44ab-b312-1777d527702c-proxy-tls") pod "isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc" (UID: "fbbed506-e8ad-44ab-b312-1777d527702c") : secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Jul 28 18:28:34.876575 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.876512    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbbed506-e8ad-44ab-b312-1777d527702c-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc\" (UID: \"fbbed506-e8ad-44ab-b312-1777d527702c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"
Jul 28 18:28:34.876751 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.876734    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbbed506-e8ad-44ab-b312-1777d527702c-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc\" (UID: \"fbbed506-e8ad-44ab-b312-1777d527702c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"
Jul 28 18:28:34.876794 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.876761    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbbed506-e8ad-44ab-b312-1777d527702c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc\" (UID: \"fbbed506-e8ad-44ab-b312-1777d527702c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"
Jul 28 18:28:34.888139 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:34.888067    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zb9n\" (UniqueName: \"kubernetes.io/projected/fbbed506-e8ad-44ab-b312-1777d527702c-kube-api-access-4zb9n\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc\" (UID: \"fbbed506-e8ad-44ab-b312-1777d527702c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"
Jul 28 18:28:35.054369 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:35.054332    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:28:35.380263 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:35.380203    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbbed506-e8ad-44ab-b312-1777d527702c-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc\" (UID: \"fbbed506-e8ad-44ab-b312-1777d527702c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"
Jul 28 18:28:35.382580 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:35.382561    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbbed506-e8ad-44ab-b312-1777d527702c-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc\" (UID: \"fbbed506-e8ad-44ab-b312-1777d527702c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"
Jul 28 18:28:35.608742 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:35.608706    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"
Jul 28 18:28:35.738379 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:35.738348    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"]
Jul 28 18:28:35.740719 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:28:35.740694    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbbed506_e8ad_44ab_b312_1777d527702c.slice/crio-13189d90d2229c3563e83f19989eab438686174589d686e05cd183d0c36030f1 WatchSource:0}: Error finding container 13189d90d2229c3563e83f19989eab438686174589d686e05cd183d0c36030f1: Status 404 returned error can't find the container with id 13189d90d2229c3563e83f19989eab438686174589d686e05cd183d0c36030f1
Jul 28 18:28:36.671376 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:36.671336    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc" event={"ID":"fbbed506-e8ad-44ab-b312-1777d527702c","Type":"ContainerStarted","Data":"32c5e80156a8337c3b2b6e0054133fd700d558ad12cb583f7ee751287cd098e4"}
Jul 28 18:28:36.671573 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:36.671405    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc" event={"ID":"fbbed506-e8ad-44ab-b312-1777d527702c","Type":"ContainerStarted","Data":"13189d90d2229c3563e83f19989eab438686174589d686e05cd183d0c36030f1"}
Jul 28 18:28:39.684115 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:39.684083    2581 generic.go:356] "Generic (PLEG): container finished" podID="fbbed506-e8ad-44ab-b312-1777d527702c" containerID="32c5e80156a8337c3b2b6e0054133fd700d558ad12cb583f7ee751287cd098e4" exitCode=0
Jul 28 18:28:39.684524 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:39.684153    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc" event={"ID":"fbbed506-e8ad-44ab-b312-1777d527702c","Type":"ContainerDied","Data":"32c5e80156a8337c3b2b6e0054133fd700d558ad12cb583f7ee751287cd098e4"}
Jul 28 18:28:40.691142 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:40.691105    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc" event={"ID":"fbbed506-e8ad-44ab-b312-1777d527702c","Type":"ContainerStarted","Data":"7c7771dcebe3cadd060c93038a15087e85171ed4a56c57f3eef264623644b58e"}
Jul 28 18:28:40.691601 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:40.691167    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc" event={"ID":"fbbed506-e8ad-44ab-b312-1777d527702c","Type":"ContainerStarted","Data":"a6bae215ca461cd715e2575bda4a0688078b7ac72bf8cd5c3d442a69cf325322"}
Jul 28 18:28:40.691601 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:40.691363    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"
Jul 28 18:28:40.713151 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:40.713103    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc" podStartSLOduration=2.946912681 podStartE2EDuration="6.713084703s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.766172022s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:28:40.71148881 +0000 UTC m=+4768.986105242" watchObservedRunningTime="2026-07-28 18:28:40.713084703 +0000 UTC m=+4768.987701135"
Jul 28 18:28:41.694397 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:41.694365    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"
Jul 28 18:28:47.703290 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:28:47.703256    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"
Jul 28 18:29:12.484011 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:12.483978    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 18:29:12.491746 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:12.491716    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 18:29:12.823204 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:12.823110    2581 scope.go:117] "RemoveContainer" containerID="060c262bf7c156d0090b5de69fc06e6c217d2bd6b7f911fb12cb6379485c3953"
Jul 28 18:29:12.833392 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:12.833362    2581 scope.go:117] "RemoveContainer" containerID="399995b214a451c86d9b60b65c374cc267119888befdd4f713472005443bb0be"
Jul 28 18:29:12.841637 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:12.841613    2581 scope.go:117] "RemoveContainer" containerID="f0999cd22603793201dea5422da610bfebcd0a2ee4f91724128bf1fcff22dc18"
Jul 28 18:29:17.708064 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:17.708030    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"
Jul 28 18:29:24.731327 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:24.731293    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"]
Jul 28 18:29:24.731904 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:24.731594    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc" podUID="fbbed506-e8ad-44ab-b312-1777d527702c" containerName="kserve-container" containerID="cri-o://a6bae215ca461cd715e2575bda4a0688078b7ac72bf8cd5c3d442a69cf325322" gracePeriod=30
Jul 28 18:29:24.731904 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:24.731684    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc" podUID="fbbed506-e8ad-44ab-b312-1777d527702c" containerName="kube-rbac-proxy" containerID="cri-o://7c7771dcebe3cadd060c93038a15087e85171ed4a56c57f3eef264623644b58e" gracePeriod=30
Jul 28 18:29:24.760643 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:24.760606    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:29:24.851589 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:24.851554    2581 generic.go:356] "Generic (PLEG): container finished" podID="fbbed506-e8ad-44ab-b312-1777d527702c" containerID="7c7771dcebe3cadd060c93038a15087e85171ed4a56c57f3eef264623644b58e" exitCode=2
Jul 28 18:29:24.851709 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:24.851605    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc" event={"ID":"fbbed506-e8ad-44ab-b312-1777d527702c","Type":"ContainerDied","Data":"7c7771dcebe3cadd060c93038a15087e85171ed4a56c57f3eef264623644b58e"}
Jul 28 18:29:27.698608 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:27.698564    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc" podUID="fbbed506-e8ad-44ab-b312-1777d527702c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.64:8643/healthz\": dial tcp 10.133.0.64:8643: connect: connection refused"
Jul 28 18:29:27.704014 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:27.703981    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc" podUID="fbbed506-e8ad-44ab-b312-1777d527702c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.64:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.133.0.64:8080: connect: connection refused"
Jul 28 18:29:29.873446 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:29.873415    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:29:29.902650 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:29.902620    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"]
Jul 28 18:29:29.905801 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:29.905783    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"
Jul 28 18:29:29.908511 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:29.908478    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Jul 28 18:29:29.908658 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:29.908583    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 28 18:29:29.916918 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:29.916891    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"]
Jul 28 18:29:30.002501 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:30.002456    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0863d75-fa9f-48e0-8a91-58928d288d2c-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5\" (UID: \"a0863d75-fa9f-48e0-8a91-58928d288d2c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"
Jul 28 18:29:30.002688 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:30.002506    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0863d75-fa9f-48e0-8a91-58928d288d2c-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5\" (UID: \"a0863d75-fa9f-48e0-8a91-58928d288d2c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"
Jul 28 18:29:30.002688 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:30.002542    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0863d75-fa9f-48e0-8a91-58928d288d2c-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5\" (UID: \"a0863d75-fa9f-48e0-8a91-58928d288d2c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"
Jul 28 18:29:30.002688 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:30.002601    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llj5x\" (UniqueName: \"kubernetes.io/projected/a0863d75-fa9f-48e0-8a91-58928d288d2c-kube-api-access-llj5x\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5\" (UID: \"a0863d75-fa9f-48e0-8a91-58928d288d2c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"
Jul 28 18:29:30.002688 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:30.002647    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0863d75-fa9f-48e0-8a91-58928d288d2c-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5\" (UID: \"a0863d75-fa9f-48e0-8a91-58928d288d2c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"
Jul 28 18:29:30.092286 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:30.092251    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:29:30.103369 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:30.103339    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0863d75-fa9f-48e0-8a91-58928d288d2c-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5\" (UID: \"a0863d75-fa9f-48e0-8a91-58928d288d2c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"
Jul 28 18:29:30.103488 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:30.103387    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0863d75-fa9f-48e0-8a91-58928d288d2c-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5\" (UID: \"a0863d75-fa9f-48e0-8a91-58928d288d2c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"
Jul 28 18:29:30.103488 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:30.103407    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0863d75-fa9f-48e0-8a91-58928d288d2c-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5\" (UID: \"a0863d75-fa9f-48e0-8a91-58928d288d2c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"
Jul 28 18:29:30.103488 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:30.103426    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0863d75-fa9f-48e0-8a91-58928d288d2c-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5\" (UID: \"a0863d75-fa9f-48e0-8a91-58928d288d2c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"
Jul 28 18:29:30.103488 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:30.103465    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llj5x\" (UniqueName: \"kubernetes.io/projected/a0863d75-fa9f-48e0-8a91-58928d288d2c-kube-api-access-llj5x\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5\" (UID: \"a0863d75-fa9f-48e0-8a91-58928d288d2c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"
Jul 28 18:29:30.103840 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:30.103817    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0863d75-fa9f-48e0-8a91-58928d288d2c-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5\" (UID: \"a0863d75-fa9f-48e0-8a91-58928d288d2c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"
Jul 28 18:29:30.104191 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:30.104157    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0863d75-fa9f-48e0-8a91-58928d288d2c-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5\" (UID: \"a0863d75-fa9f-48e0-8a91-58928d288d2c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"
Jul 28 18:29:30.104191 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:30.104186    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0863d75-fa9f-48e0-8a91-58928d288d2c-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5\" (UID: \"a0863d75-fa9f-48e0-8a91-58928d288d2c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"
Jul 28 18:29:30.106018 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:30.105996    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0863d75-fa9f-48e0-8a91-58928d288d2c-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5\" (UID: \"a0863d75-fa9f-48e0-8a91-58928d288d2c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"
Jul 28 18:29:30.113275 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:30.113246    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llj5x\" (UniqueName: \"kubernetes.io/projected/a0863d75-fa9f-48e0-8a91-58928d288d2c-kube-api-access-llj5x\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5\" (UID: \"a0863d75-fa9f-48e0-8a91-58928d288d2c\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"
Jul 28 18:29:30.217922 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:30.217882    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"
Jul 28 18:29:30.353701 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:30.353672    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"]
Jul 28 18:29:30.356165 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:29:30.356120    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0863d75_fa9f_48e0_8a91_58928d288d2c.slice/crio-f9c2717b9198756bceff6c8a1a2f4b8ef85c4b51fc271ce894d2fb1915ecfd01 WatchSource:0}: Error finding container f9c2717b9198756bceff6c8a1a2f4b8ef85c4b51fc271ce894d2fb1915ecfd01: Status 404 returned error can't find the container with id f9c2717b9198756bceff6c8a1a2f4b8ef85c4b51fc271ce894d2fb1915ecfd01
Jul 28 18:29:30.875677 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:30.875631    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5" event={"ID":"a0863d75-fa9f-48e0-8a91-58928d288d2c","Type":"ContainerStarted","Data":"53c4e11cb5ae7ed188d145969a2c16d0fc2295a8b800e2f8bff7543a21244260"}
Jul 28 18:29:30.876212 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:30.875698    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5" event={"ID":"a0863d75-fa9f-48e0-8a91-58928d288d2c","Type":"ContainerStarted","Data":"f9c2717b9198756bceff6c8a1a2f4b8ef85c4b51fc271ce894d2fb1915ecfd01"}
Jul 28 18:29:31.674894 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.674835    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"
Jul 28 18:29:31.716141 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.716106    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbbed506-e8ad-44ab-b312-1777d527702c-kserve-provision-location\") pod \"fbbed506-e8ad-44ab-b312-1777d527702c\" (UID: \"fbbed506-e8ad-44ab-b312-1777d527702c\") "
Jul 28 18:29:31.716141 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.716143    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbbed506-e8ad-44ab-b312-1777d527702c-proxy-tls\") pod \"fbbed506-e8ad-44ab-b312-1777d527702c\" (UID: \"fbbed506-e8ad-44ab-b312-1777d527702c\") "
Jul 28 18:29:31.716368 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.716174    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbbed506-e8ad-44ab-b312-1777d527702c-kube-rbac-proxy-sar-config\") pod \"fbbed506-e8ad-44ab-b312-1777d527702c\" (UID: \"fbbed506-e8ad-44ab-b312-1777d527702c\") "
Jul 28 18:29:31.716368 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.716209    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zb9n\" (UniqueName: \"kubernetes.io/projected/fbbed506-e8ad-44ab-b312-1777d527702c-kube-api-access-4zb9n\") pod \"fbbed506-e8ad-44ab-b312-1777d527702c\" (UID: \"fbbed506-e8ad-44ab-b312-1777d527702c\") "
Jul 28 18:29:31.716368 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.716239    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbbed506-e8ad-44ab-b312-1777d527702c-cabundle-cert\") pod \"fbbed506-e8ad-44ab-b312-1777d527702c\" (UID: \"fbbed506-e8ad-44ab-b312-1777d527702c\") "
Jul 28 18:29:31.716597 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.716501    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbbed506-e8ad-44ab-b312-1777d527702c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fbbed506-e8ad-44ab-b312-1777d527702c" (UID: "fbbed506-e8ad-44ab-b312-1777d527702c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:29:31.716659 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.716609    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbbed506-e8ad-44ab-b312-1777d527702c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fbbed506-e8ad-44ab-b312-1777d527702c" (UID: "fbbed506-e8ad-44ab-b312-1777d527702c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:29:31.716715 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.716691    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbbed506-e8ad-44ab-b312-1777d527702c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fbbed506-e8ad-44ab-b312-1777d527702c" (UID: "fbbed506-e8ad-44ab-b312-1777d527702c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:29:31.718329 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.718299    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbbed506-e8ad-44ab-b312-1777d527702c-kube-api-access-4zb9n" (OuterVolumeSpecName: "kube-api-access-4zb9n") pod "fbbed506-e8ad-44ab-b312-1777d527702c" (UID: "fbbed506-e8ad-44ab-b312-1777d527702c"). InnerVolumeSpecName "kube-api-access-4zb9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:29:31.718329 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.718296    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbbed506-e8ad-44ab-b312-1777d527702c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fbbed506-e8ad-44ab-b312-1777d527702c" (UID: "fbbed506-e8ad-44ab-b312-1777d527702c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:29:31.817205 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.817165    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fbbed506-e8ad-44ab-b312-1777d527702c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:29:31.817205 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.817197    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zb9n\" (UniqueName: \"kubernetes.io/projected/fbbed506-e8ad-44ab-b312-1777d527702c-kube-api-access-4zb9n\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:29:31.817205 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.817207    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fbbed506-e8ad-44ab-b312-1777d527702c-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:29:31.817205 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.817217    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fbbed506-e8ad-44ab-b312-1777d527702c-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:29:31.817477 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.817227    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fbbed506-e8ad-44ab-b312-1777d527702c-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:29:31.880869 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.880835    2581 generic.go:356] "Generic (PLEG): container finished" podID="fbbed506-e8ad-44ab-b312-1777d527702c" containerID="a6bae215ca461cd715e2575bda4a0688078b7ac72bf8cd5c3d442a69cf325322" exitCode=0
Jul 28 18:29:31.881376 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.880905    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"
Jul 28 18:29:31.881376 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.880918    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc" event={"ID":"fbbed506-e8ad-44ab-b312-1777d527702c","Type":"ContainerDied","Data":"a6bae215ca461cd715e2575bda4a0688078b7ac72bf8cd5c3d442a69cf325322"}
Jul 28 18:29:31.881376 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.880992    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc" event={"ID":"fbbed506-e8ad-44ab-b312-1777d527702c","Type":"ContainerDied","Data":"13189d90d2229c3563e83f19989eab438686174589d686e05cd183d0c36030f1"}
Jul 28 18:29:31.881376 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.881021    2581 scope.go:117] "RemoveContainer" containerID="7c7771dcebe3cadd060c93038a15087e85171ed4a56c57f3eef264623644b58e"
Jul 28 18:29:31.889426 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.889405    2581 scope.go:117] "RemoveContainer" containerID="a6bae215ca461cd715e2575bda4a0688078b7ac72bf8cd5c3d442a69cf325322"
Jul 28 18:29:31.896857 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.896836    2581 scope.go:117] "RemoveContainer" containerID="32c5e80156a8337c3b2b6e0054133fd700d558ad12cb583f7ee751287cd098e4"
Jul 28 18:29:31.904488 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.904458    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"]
Jul 28 18:29:31.905075 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.905048    2581 scope.go:117] "RemoveContainer" containerID="7c7771dcebe3cadd060c93038a15087e85171ed4a56c57f3eef264623644b58e"
Jul 28 18:29:31.905418 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:29:31.905395    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c7771dcebe3cadd060c93038a15087e85171ed4a56c57f3eef264623644b58e\": container with ID starting with 7c7771dcebe3cadd060c93038a15087e85171ed4a56c57f3eef264623644b58e not found: ID does not exist" containerID="7c7771dcebe3cadd060c93038a15087e85171ed4a56c57f3eef264623644b58e"
Jul 28 18:29:31.905509 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.905428    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c7771dcebe3cadd060c93038a15087e85171ed4a56c57f3eef264623644b58e"} err="failed to get container status \"7c7771dcebe3cadd060c93038a15087e85171ed4a56c57f3eef264623644b58e\": rpc error: code = NotFound desc = could not find container \"7c7771dcebe3cadd060c93038a15087e85171ed4a56c57f3eef264623644b58e\": container with ID starting with 7c7771dcebe3cadd060c93038a15087e85171ed4a56c57f3eef264623644b58e not found: ID does not exist"
Jul 28 18:29:31.905509 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.905448    2581 scope.go:117] "RemoveContainer" containerID="a6bae215ca461cd715e2575bda4a0688078b7ac72bf8cd5c3d442a69cf325322"
Jul 28 18:29:31.905737 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:29:31.905710    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6bae215ca461cd715e2575bda4a0688078b7ac72bf8cd5c3d442a69cf325322\": container with ID starting with a6bae215ca461cd715e2575bda4a0688078b7ac72bf8cd5c3d442a69cf325322 not found: ID does not exist" containerID="a6bae215ca461cd715e2575bda4a0688078b7ac72bf8cd5c3d442a69cf325322"
Jul 28 18:29:31.905829 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.905769    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6bae215ca461cd715e2575bda4a0688078b7ac72bf8cd5c3d442a69cf325322"} err="failed to get container status \"a6bae215ca461cd715e2575bda4a0688078b7ac72bf8cd5c3d442a69cf325322\": rpc error: code = NotFound desc = could not find container \"a6bae215ca461cd715e2575bda4a0688078b7ac72bf8cd5c3d442a69cf325322\": container with ID starting with a6bae215ca461cd715e2575bda4a0688078b7ac72bf8cd5c3d442a69cf325322 not found: ID does not exist"
Jul 28 18:29:31.905829 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.905795    2581 scope.go:117] "RemoveContainer" containerID="32c5e80156a8337c3b2b6e0054133fd700d558ad12cb583f7ee751287cd098e4"
Jul 28 18:29:31.906166 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:29:31.906146    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32c5e80156a8337c3b2b6e0054133fd700d558ad12cb583f7ee751287cd098e4\": container with ID starting with 32c5e80156a8337c3b2b6e0054133fd700d558ad12cb583f7ee751287cd098e4 not found: ID does not exist" containerID="32c5e80156a8337c3b2b6e0054133fd700d558ad12cb583f7ee751287cd098e4"
Jul 28 18:29:31.906246 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.906173    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32c5e80156a8337c3b2b6e0054133fd700d558ad12cb583f7ee751287cd098e4"} err="failed to get container status \"32c5e80156a8337c3b2b6e0054133fd700d558ad12cb583f7ee751287cd098e4\": rpc error: code = NotFound desc = could not find container \"32c5e80156a8337c3b2b6e0054133fd700d558ad12cb583f7ee751287cd098e4\": container with ID starting with 32c5e80156a8337c3b2b6e0054133fd700d558ad12cb583f7ee751287cd098e4 not found: ID does not exist"
Jul 28 18:29:31.907160 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:31.907142    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-jwvxc"]
Jul 28 18:29:32.289503 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:32.289471    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbbed506-e8ad-44ab-b312-1777d527702c" path="/var/lib/kubelet/pods/fbbed506-e8ad-44ab-b312-1777d527702c/volumes"
Jul 28 18:29:34.893053 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:34.892969    2581 generic.go:356] "Generic (PLEG): container finished" podID="a0863d75-fa9f-48e0-8a91-58928d288d2c" containerID="53c4e11cb5ae7ed188d145969a2c16d0fc2295a8b800e2f8bff7543a21244260" exitCode=0
Jul 28 18:29:34.893409 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:34.893044    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5" event={"ID":"a0863d75-fa9f-48e0-8a91-58928d288d2c","Type":"ContainerDied","Data":"53c4e11cb5ae7ed188d145969a2c16d0fc2295a8b800e2f8bff7543a21244260"}
Jul 28 18:29:35.898830 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:35.898792    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5" event={"ID":"a0863d75-fa9f-48e0-8a91-58928d288d2c","Type":"ContainerStarted","Data":"a860c612c3cf574f082a224c26acd606cca8c384b372f4b64846ac609e7de59c"}
Jul 28 18:29:35.899273 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:35.898881    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5" event={"ID":"a0863d75-fa9f-48e0-8a91-58928d288d2c","Type":"ContainerStarted","Data":"848e6c53ca83216c35b8cb0292609b9feeebdfbb0dd0ba20660595f5e0eeac3f"}
Jul 28 18:29:35.899273 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:35.899121    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"
Jul 28 18:29:35.899347 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:35.899288    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"
Jul 28 18:29:35.919979 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:35.919919    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5" podStartSLOduration=2.883746468 podStartE2EDuration="6.919906369s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.036159901s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:29:35.919092458 +0000 UTC m=+4824.193708888" watchObservedRunningTime="2026-07-28 18:29:35.919906369 +0000 UTC m=+4824.194522801"
Jul 28 18:29:41.907685 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:29:41.907652    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"
Jul 28 18:30:11.912206 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:11.912123    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"
Jul 28 18:30:20.005872 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:20.005835    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"]
Jul 28 18:30:20.006317 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:20.006112    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5" podUID="a0863d75-fa9f-48e0-8a91-58928d288d2c" containerName="kserve-container" containerID="cri-o://848e6c53ca83216c35b8cb0292609b9feeebdfbb0dd0ba20660595f5e0eeac3f" gracePeriod=30
Jul 28 18:30:20.006317 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:20.006165    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5" podUID="a0863d75-fa9f-48e0-8a91-58928d288d2c" containerName="kube-rbac-proxy" containerID="cri-o://a860c612c3cf574f082a224c26acd606cca8c384b372f4b64846ac609e7de59c" gracePeriod=30
Jul 28 18:30:20.034357 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:20.034290    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:30:21.057264 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:21.057228    2581 generic.go:356] "Generic (PLEG): container finished" podID="a0863d75-fa9f-48e0-8a91-58928d288d2c" containerID="a860c612c3cf574f082a224c26acd606cca8c384b372f4b64846ac609e7de59c" exitCode=2
Jul 28 18:30:21.057658 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:21.057305    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5" event={"ID":"a0863d75-fa9f-48e0-8a91-58928d288d2c","Type":"ContainerDied","Data":"a860c612c3cf574f082a224c26acd606cca8c384b372f4b64846ac609e7de59c"}
Jul 28 18:30:21.903248 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:21.903210    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5" podUID="a0863d75-fa9f-48e0-8a91-58928d288d2c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.65:8643/healthz\": dial tcp 10.133.0.65:8643: connect: connection refused"
Jul 28 18:30:25.222796 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.222747    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:30:25.287934 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.287901    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"]
Jul 28 18:30:25.288269 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.288251    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbbed506-e8ad-44ab-b312-1777d527702c" containerName="storage-initializer"
Jul 28 18:30:25.288320 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.288271    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbbed506-e8ad-44ab-b312-1777d527702c" containerName="storage-initializer"
Jul 28 18:30:25.288320 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.288281    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbbed506-e8ad-44ab-b312-1777d527702c" containerName="kserve-container"
Jul 28 18:30:25.288320 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.288287    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbbed506-e8ad-44ab-b312-1777d527702c" containerName="kserve-container"
Jul 28 18:30:25.288320 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.288303    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbbed506-e8ad-44ab-b312-1777d527702c" containerName="kube-rbac-proxy"
Jul 28 18:30:25.288320 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.288309    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbbed506-e8ad-44ab-b312-1777d527702c" containerName="kube-rbac-proxy"
Jul 28 18:30:25.288487 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.288359    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbbed506-e8ad-44ab-b312-1777d527702c" containerName="kserve-container"
Jul 28 18:30:25.288487 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.288368    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbbed506-e8ad-44ab-b312-1777d527702c" containerName="kube-rbac-proxy"
Jul 28 18:30:25.291492 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.291475    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"
Jul 28 18:30:25.294835 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.294813    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Jul 28 18:30:25.295295 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.295280    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Jul 28 18:30:25.313244 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.313216    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"]
Jul 28 18:30:25.411925 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.411880    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b7dad16-d1eb-463b-abf8-f5a51254d464-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2w7rt\" (UID: \"2b7dad16-d1eb-463b-abf8-f5a51254d464\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"
Jul 28 18:30:25.411925 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.411922    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b7dad16-d1eb-463b-abf8-f5a51254d464-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2w7rt\" (UID: \"2b7dad16-d1eb-463b-abf8-f5a51254d464\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"
Jul 28 18:30:25.412216 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.411973    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b7dad16-d1eb-463b-abf8-f5a51254d464-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2w7rt\" (UID: \"2b7dad16-d1eb-463b-abf8-f5a51254d464\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"
Jul 28 18:30:25.412216 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.412096    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gccxm\" (UniqueName: \"kubernetes.io/projected/2b7dad16-d1eb-463b-abf8-f5a51254d464-kube-api-access-gccxm\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2w7rt\" (UID: \"2b7dad16-d1eb-463b-abf8-f5a51254d464\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"
Jul 28 18:30:25.412216 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.412150    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b7dad16-d1eb-463b-abf8-f5a51254d464-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2w7rt\" (UID: \"2b7dad16-d1eb-463b-abf8-f5a51254d464\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"
Jul 28 18:30:25.434521 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.434472    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:30:25.513025 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.512898    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b7dad16-d1eb-463b-abf8-f5a51254d464-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2w7rt\" (UID: \"2b7dad16-d1eb-463b-abf8-f5a51254d464\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"
Jul 28 18:30:25.513025 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.512965    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b7dad16-d1eb-463b-abf8-f5a51254d464-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2w7rt\" (UID: \"2b7dad16-d1eb-463b-abf8-f5a51254d464\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"
Jul 28 18:30:25.513025 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.513003    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b7dad16-d1eb-463b-abf8-f5a51254d464-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2w7rt\" (UID: \"2b7dad16-d1eb-463b-abf8-f5a51254d464\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"
Jul 28 18:30:25.513330 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.513055    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gccxm\" (UniqueName: \"kubernetes.io/projected/2b7dad16-d1eb-463b-abf8-f5a51254d464-kube-api-access-gccxm\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2w7rt\" (UID: \"2b7dad16-d1eb-463b-abf8-f5a51254d464\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"
Jul 28 18:30:25.513330 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.513094    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b7dad16-d1eb-463b-abf8-f5a51254d464-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2w7rt\" (UID: \"2b7dad16-d1eb-463b-abf8-f5a51254d464\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"
Jul 28 18:30:25.513539 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.513515    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b7dad16-d1eb-463b-abf8-f5a51254d464-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2w7rt\" (UID: \"2b7dad16-d1eb-463b-abf8-f5a51254d464\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"
Jul 28 18:30:25.513725 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.513705    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b7dad16-d1eb-463b-abf8-f5a51254d464-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2w7rt\" (UID: \"2b7dad16-d1eb-463b-abf8-f5a51254d464\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"
Jul 28 18:30:25.513766 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.513726    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b7dad16-d1eb-463b-abf8-f5a51254d464-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2w7rt\" (UID: \"2b7dad16-d1eb-463b-abf8-f5a51254d464\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"
Jul 28 18:30:25.515548 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.515530    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b7dad16-d1eb-463b-abf8-f5a51254d464-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2w7rt\" (UID: \"2b7dad16-d1eb-463b-abf8-f5a51254d464\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"
Jul 28 18:30:25.524194 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.524160    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gccxm\" (UniqueName: \"kubernetes.io/projected/2b7dad16-d1eb-463b-abf8-f5a51254d464-kube-api-access-gccxm\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2w7rt\" (UID: \"2b7dad16-d1eb-463b-abf8-f5a51254d464\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"
Jul 28 18:30:25.601712 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.601667    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"
Jul 28 18:30:25.738614 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:25.738579    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"]
Jul 28 18:30:25.741525 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:30:25.741490    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7dad16_d1eb_463b_abf8_f5a51254d464.slice/crio-e61feb35e046e1ec7e13cefcaedd45773f1bb3430e9309d48e5ef9597522ea38 WatchSource:0}: Error finding container e61feb35e046e1ec7e13cefcaedd45773f1bb3430e9309d48e5ef9597522ea38: Status 404 returned error can't find the container with id e61feb35e046e1ec7e13cefcaedd45773f1bb3430e9309d48e5ef9597522ea38
Jul 28 18:30:26.076798 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:26.076706    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt" event={"ID":"2b7dad16-d1eb-463b-abf8-f5a51254d464","Type":"ContainerStarted","Data":"55b738cfb1125d6ee2f0554130dc889af18bdce8269ab4df50ba896cc0b9ae30"}
Jul 28 18:30:26.076798 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:26.076778    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt" event={"ID":"2b7dad16-d1eb-463b-abf8-f5a51254d464","Type":"ContainerStarted","Data":"e61feb35e046e1ec7e13cefcaedd45773f1bb3430e9309d48e5ef9597522ea38"}
Jul 28 18:30:26.748235 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:26.748207    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"
Jul 28 18:30:26.824071 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:26.823983    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0863d75-fa9f-48e0-8a91-58928d288d2c-kube-rbac-proxy-sar-config\") pod \"a0863d75-fa9f-48e0-8a91-58928d288d2c\" (UID: \"a0863d75-fa9f-48e0-8a91-58928d288d2c\") "
Jul 28 18:30:26.824071 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:26.824043    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0863d75-fa9f-48e0-8a91-58928d288d2c-cabundle-cert\") pod \"a0863d75-fa9f-48e0-8a91-58928d288d2c\" (UID: \"a0863d75-fa9f-48e0-8a91-58928d288d2c\") "
Jul 28 18:30:26.824071 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:26.824066    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0863d75-fa9f-48e0-8a91-58928d288d2c-proxy-tls\") pod \"a0863d75-fa9f-48e0-8a91-58928d288d2c\" (UID: \"a0863d75-fa9f-48e0-8a91-58928d288d2c\") "
Jul 28 18:30:26.824321 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:26.824101    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0863d75-fa9f-48e0-8a91-58928d288d2c-kserve-provision-location\") pod \"a0863d75-fa9f-48e0-8a91-58928d288d2c\" (UID: \"a0863d75-fa9f-48e0-8a91-58928d288d2c\") "
Jul 28 18:30:26.824321 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:26.824123    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llj5x\" (UniqueName: \"kubernetes.io/projected/a0863d75-fa9f-48e0-8a91-58928d288d2c-kube-api-access-llj5x\") pod \"a0863d75-fa9f-48e0-8a91-58928d288d2c\" (UID: \"a0863d75-fa9f-48e0-8a91-58928d288d2c\") "
Jul 28 18:30:26.824468 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:26.824433    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0863d75-fa9f-48e0-8a91-58928d288d2c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a0863d75-fa9f-48e0-8a91-58928d288d2c" (UID: "a0863d75-fa9f-48e0-8a91-58928d288d2c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:30:26.824468 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:26.824443    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0863d75-fa9f-48e0-8a91-58928d288d2c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a0863d75-fa9f-48e0-8a91-58928d288d2c" (UID: "a0863d75-fa9f-48e0-8a91-58928d288d2c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:30:26.824555 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:26.824443    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0863d75-fa9f-48e0-8a91-58928d288d2c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a0863d75-fa9f-48e0-8a91-58928d288d2c" (UID: "a0863d75-fa9f-48e0-8a91-58928d288d2c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:30:26.826356 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:26.826322    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0863d75-fa9f-48e0-8a91-58928d288d2c-kube-api-access-llj5x" (OuterVolumeSpecName: "kube-api-access-llj5x") pod "a0863d75-fa9f-48e0-8a91-58928d288d2c" (UID: "a0863d75-fa9f-48e0-8a91-58928d288d2c"). InnerVolumeSpecName "kube-api-access-llj5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:30:26.826480 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:26.826385    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0863d75-fa9f-48e0-8a91-58928d288d2c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a0863d75-fa9f-48e0-8a91-58928d288d2c" (UID: "a0863d75-fa9f-48e0-8a91-58928d288d2c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:30:26.925163 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:26.925102    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a0863d75-fa9f-48e0-8a91-58928d288d2c-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:30:26.925163 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:26.925154    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0863d75-fa9f-48e0-8a91-58928d288d2c-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:30:26.925163 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:26.925167    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a0863d75-fa9f-48e0-8a91-58928d288d2c-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:30:26.925163 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:26.925177    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llj5x\" (UniqueName: \"kubernetes.io/projected/a0863d75-fa9f-48e0-8a91-58928d288d2c-kube-api-access-llj5x\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:30:26.925163 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:26.925186    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a0863d75-fa9f-48e0-8a91-58928d288d2c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:30:27.083504 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:27.083406    2581 generic.go:356] "Generic (PLEG): container finished" podID="a0863d75-fa9f-48e0-8a91-58928d288d2c" containerID="848e6c53ca83216c35b8cb0292609b9feeebdfbb0dd0ba20660595f5e0eeac3f" exitCode=0
Jul 28 18:30:27.083504 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:27.083469    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"
Jul 28 18:30:27.083764 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:27.083498    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5" event={"ID":"a0863d75-fa9f-48e0-8a91-58928d288d2c","Type":"ContainerDied","Data":"848e6c53ca83216c35b8cb0292609b9feeebdfbb0dd0ba20660595f5e0eeac3f"}
Jul 28 18:30:27.083764 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:27.083577    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5" event={"ID":"a0863d75-fa9f-48e0-8a91-58928d288d2c","Type":"ContainerDied","Data":"f9c2717b9198756bceff6c8a1a2f4b8ef85c4b51fc271ce894d2fb1915ecfd01"}
Jul 28 18:30:27.083764 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:27.083614    2581 scope.go:117] "RemoveContainer" containerID="a860c612c3cf574f082a224c26acd606cca8c384b372f4b64846ac609e7de59c"
Jul 28 18:30:27.092418 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:27.092398    2581 scope.go:117] "RemoveContainer" containerID="848e6c53ca83216c35b8cb0292609b9feeebdfbb0dd0ba20660595f5e0eeac3f"
Jul 28 18:30:27.100492 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:27.100463    2581 scope.go:117] "RemoveContainer" containerID="53c4e11cb5ae7ed188d145969a2c16d0fc2295a8b800e2f8bff7543a21244260"
Jul 28 18:30:27.106739 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:27.106707    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"]
Jul 28 18:30:27.109648 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:27.109620    2581 scope.go:117] "RemoveContainer" containerID="a860c612c3cf574f082a224c26acd606cca8c384b372f4b64846ac609e7de59c"
Jul 28 18:30:27.110595 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:30:27.110562    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a860c612c3cf574f082a224c26acd606cca8c384b372f4b64846ac609e7de59c\": container with ID starting with a860c612c3cf574f082a224c26acd606cca8c384b372f4b64846ac609e7de59c not found: ID does not exist" containerID="a860c612c3cf574f082a224c26acd606cca8c384b372f4b64846ac609e7de59c"
Jul 28 18:30:27.110727 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:27.110608    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a860c612c3cf574f082a224c26acd606cca8c384b372f4b64846ac609e7de59c"} err="failed to get container status \"a860c612c3cf574f082a224c26acd606cca8c384b372f4b64846ac609e7de59c\": rpc error: code = NotFound desc = could not find container \"a860c612c3cf574f082a224c26acd606cca8c384b372f4b64846ac609e7de59c\": container with ID starting with a860c612c3cf574f082a224c26acd606cca8c384b372f4b64846ac609e7de59c not found: ID does not exist"
Jul 28 18:30:27.110727 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:27.110635    2581 scope.go:117] "RemoveContainer" containerID="848e6c53ca83216c35b8cb0292609b9feeebdfbb0dd0ba20660595f5e0eeac3f"
Jul 28 18:30:27.110994 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:30:27.110971    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"848e6c53ca83216c35b8cb0292609b9feeebdfbb0dd0ba20660595f5e0eeac3f\": container with ID starting with 848e6c53ca83216c35b8cb0292609b9feeebdfbb0dd0ba20660595f5e0eeac3f not found: ID does not exist" containerID="848e6c53ca83216c35b8cb0292609b9feeebdfbb0dd0ba20660595f5e0eeac3f"
Jul 28 18:30:27.111045 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:27.111003    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"848e6c53ca83216c35b8cb0292609b9feeebdfbb0dd0ba20660595f5e0eeac3f"} err="failed to get container status \"848e6c53ca83216c35b8cb0292609b9feeebdfbb0dd0ba20660595f5e0eeac3f\": rpc error: code = NotFound desc = could not find container \"848e6c53ca83216c35b8cb0292609b9feeebdfbb0dd0ba20660595f5e0eeac3f\": container with ID starting with 848e6c53ca83216c35b8cb0292609b9feeebdfbb0dd0ba20660595f5e0eeac3f not found: ID does not exist"
Jul 28 18:30:27.111045 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:27.111021    2581 scope.go:117] "RemoveContainer" containerID="53c4e11cb5ae7ed188d145969a2c16d0fc2295a8b800e2f8bff7543a21244260"
Jul 28 18:30:27.111310 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:30:27.111290    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53c4e11cb5ae7ed188d145969a2c16d0fc2295a8b800e2f8bff7543a21244260\": container with ID starting with 53c4e11cb5ae7ed188d145969a2c16d0fc2295a8b800e2f8bff7543a21244260 not found: ID does not exist" containerID="53c4e11cb5ae7ed188d145969a2c16d0fc2295a8b800e2f8bff7543a21244260"
Jul 28 18:30:27.111354 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:27.111322    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53c4e11cb5ae7ed188d145969a2c16d0fc2295a8b800e2f8bff7543a21244260"} err="failed to get container status \"53c4e11cb5ae7ed188d145969a2c16d0fc2295a8b800e2f8bff7543a21244260\": rpc error: code = NotFound desc = could not find container \"53c4e11cb5ae7ed188d145969a2c16d0fc2295a8b800e2f8bff7543a21244260\": container with ID starting with 53c4e11cb5ae7ed188d145969a2c16d0fc2295a8b800e2f8bff7543a21244260 not found: ID does not exist"
Jul 28 18:30:27.111713 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:27.111693    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-bkxh5"]
Jul 28 18:30:28.289471 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:28.289440    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0863d75-fa9f-48e0-8a91-58928d288d2c" path="/var/lib/kubelet/pods/a0863d75-fa9f-48e0-8a91-58928d288d2c/volumes"
Jul 28 18:30:30.096695 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:30.096655    2581 generic.go:356] "Generic (PLEG): container finished" podID="2b7dad16-d1eb-463b-abf8-f5a51254d464" containerID="55b738cfb1125d6ee2f0554130dc889af18bdce8269ab4df50ba896cc0b9ae30" exitCode=0
Jul 28 18:30:30.097118 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:30.096705    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt" event={"ID":"2b7dad16-d1eb-463b-abf8-f5a51254d464","Type":"ContainerDied","Data":"55b738cfb1125d6ee2f0554130dc889af18bdce8269ab4df50ba896cc0b9ae30"}
Jul 28 18:30:31.102258 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:31.102223    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt" event={"ID":"2b7dad16-d1eb-463b-abf8-f5a51254d464","Type":"ContainerStarted","Data":"166c4e0f735156c078f79ac2559b691d607ea6dd98be8b555a34f363f12a53c9"}
Jul 28 18:30:31.102705 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:31.102285    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt" event={"ID":"2b7dad16-d1eb-463b-abf8-f5a51254d464","Type":"ContainerStarted","Data":"f3cfe506f9dc3f4726beee9cad786b32451f4f13125ec40a63c1a6a0d6002ee8"}
Jul 28 18:30:31.102705 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:31.102572    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"
Jul 28 18:30:31.102821 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:31.102732    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"
Jul 28 18:30:31.104066 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:31.104039    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt" podUID="2b7dad16-d1eb-463b-abf8-f5a51254d464" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 28 18:30:31.124185 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:31.124132    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt" podStartSLOduration=2.447422174 podStartE2EDuration="6.12411519s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.676693016s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:30:31.122381304 +0000 UTC m=+4879.396997737" watchObservedRunningTime="2026-07-28 18:30:31.12411519 +0000 UTC m=+4879.398731621"
Jul 28 18:30:32.106255 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:32.106215    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt" podUID="2b7dad16-d1eb-463b-abf8-f5a51254d464" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 28 18:30:32.972375 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:32.972340    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:30:33.109836 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:33.109796    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt" podUID="2b7dad16-d1eb-463b-abf8-f5a51254d464" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 28 18:30:38.114633 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:38.114603    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"
Jul 28 18:30:38.115289 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:38.115261    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt" podUID="2b7dad16-d1eb-463b-abf8-f5a51254d464" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 28 18:30:48.115440 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:48.115398    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt" podUID="2b7dad16-d1eb-463b-abf8-f5a51254d464" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 28 18:30:58.115344 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:30:58.115291    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt" podUID="2b7dad16-d1eb-463b-abf8-f5a51254d464" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 28 18:31:08.115891 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:08.115849    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt" podUID="2b7dad16-d1eb-463b-abf8-f5a51254d464" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 28 18:31:18.115872 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:18.115828    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt" podUID="2b7dad16-d1eb-463b-abf8-f5a51254d464" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 28 18:31:28.115689 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:28.115638    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt" podUID="2b7dad16-d1eb-463b-abf8-f5a51254d464" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 28 18:31:38.116150 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:38.116069    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"
Jul 28 18:31:45.277175 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:45.277134    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"]
Jul 28 18:31:45.277583 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:45.277449    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt" podUID="2b7dad16-d1eb-463b-abf8-f5a51254d464" containerName="kserve-container" containerID="cri-o://f3cfe506f9dc3f4726beee9cad786b32451f4f13125ec40a63c1a6a0d6002ee8" gracePeriod=30
Jul 28 18:31:45.277583 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:45.277480    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt" podUID="2b7dad16-d1eb-463b-abf8-f5a51254d464" containerName="kube-rbac-proxy" containerID="cri-o://166c4e0f735156c078f79ac2559b691d607ea6dd98be8b555a34f363f12a53c9" gracePeriod=30
Jul 28 18:31:45.301081 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:45.301025    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:31:46.365040 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:46.365007    2581 generic.go:356] "Generic (PLEG): container finished" podID="2b7dad16-d1eb-463b-abf8-f5a51254d464" containerID="166c4e0f735156c078f79ac2559b691d607ea6dd98be8b555a34f363f12a53c9" exitCode=2
Jul 28 18:31:46.365436 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:46.365081    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt" event={"ID":"2b7dad16-d1eb-463b-abf8-f5a51254d464","Type":"ContainerDied","Data":"166c4e0f735156c078f79ac2559b691d607ea6dd98be8b555a34f363f12a53c9"}
Jul 28 18:31:48.110444 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:48.110402    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt" podUID="2b7dad16-d1eb-463b-abf8-f5a51254d464" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.66:8643/healthz\": dial tcp 10.133.0.66:8643: connect: connection refused"
Jul 28 18:31:48.116025 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:48.115988    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt" podUID="2b7dad16-d1eb-463b-abf8-f5a51254d464" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 28 18:31:49.033772 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.033743    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"
Jul 28 18:31:49.118262 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.118160    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b7dad16-d1eb-463b-abf8-f5a51254d464-kube-rbac-proxy-sar-config\") pod \"2b7dad16-d1eb-463b-abf8-f5a51254d464\" (UID: \"2b7dad16-d1eb-463b-abf8-f5a51254d464\") "
Jul 28 18:31:49.118262 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.118213    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b7dad16-d1eb-463b-abf8-f5a51254d464-kserve-provision-location\") pod \"2b7dad16-d1eb-463b-abf8-f5a51254d464\" (UID: \"2b7dad16-d1eb-463b-abf8-f5a51254d464\") "
Jul 28 18:31:49.118262 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.118240    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b7dad16-d1eb-463b-abf8-f5a51254d464-proxy-tls\") pod \"2b7dad16-d1eb-463b-abf8-f5a51254d464\" (UID: \"2b7dad16-d1eb-463b-abf8-f5a51254d464\") "
Jul 28 18:31:49.118767 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.118506    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b7dad16-d1eb-463b-abf8-f5a51254d464-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2b7dad16-d1eb-463b-abf8-f5a51254d464" (UID: "2b7dad16-d1eb-463b-abf8-f5a51254d464"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:31:49.118767 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.118542    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b7dad16-d1eb-463b-abf8-f5a51254d464-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2b7dad16-d1eb-463b-abf8-f5a51254d464" (UID: "2b7dad16-d1eb-463b-abf8-f5a51254d464"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:31:49.120294 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.120272    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b7dad16-d1eb-463b-abf8-f5a51254d464-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2b7dad16-d1eb-463b-abf8-f5a51254d464" (UID: "2b7dad16-d1eb-463b-abf8-f5a51254d464"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:31:49.218689 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.218644    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b7dad16-d1eb-463b-abf8-f5a51254d464-cabundle-cert\") pod \"2b7dad16-d1eb-463b-abf8-f5a51254d464\" (UID: \"2b7dad16-d1eb-463b-abf8-f5a51254d464\") "
Jul 28 18:31:49.218689 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.218690    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gccxm\" (UniqueName: \"kubernetes.io/projected/2b7dad16-d1eb-463b-abf8-f5a51254d464-kube-api-access-gccxm\") pod \"2b7dad16-d1eb-463b-abf8-f5a51254d464\" (UID: \"2b7dad16-d1eb-463b-abf8-f5a51254d464\") "
Jul 28 18:31:49.218980 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.218885    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2b7dad16-d1eb-463b-abf8-f5a51254d464-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:31:49.218980 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.218901    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2b7dad16-d1eb-463b-abf8-f5a51254d464-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:31:49.218980 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.218915    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2b7dad16-d1eb-463b-abf8-f5a51254d464-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:31:49.219142 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.219102    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b7dad16-d1eb-463b-abf8-f5a51254d464-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2b7dad16-d1eb-463b-abf8-f5a51254d464" (UID: "2b7dad16-d1eb-463b-abf8-f5a51254d464"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:31:49.220849 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.220817    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b7dad16-d1eb-463b-abf8-f5a51254d464-kube-api-access-gccxm" (OuterVolumeSpecName: "kube-api-access-gccxm") pod "2b7dad16-d1eb-463b-abf8-f5a51254d464" (UID: "2b7dad16-d1eb-463b-abf8-f5a51254d464"). InnerVolumeSpecName "kube-api-access-gccxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:31:49.319428 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.319390    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2b7dad16-d1eb-463b-abf8-f5a51254d464-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:31:49.319428 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.319422    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gccxm\" (UniqueName: \"kubernetes.io/projected/2b7dad16-d1eb-463b-abf8-f5a51254d464-kube-api-access-gccxm\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:31:49.376772 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.376678    2581 generic.go:356] "Generic (PLEG): container finished" podID="2b7dad16-d1eb-463b-abf8-f5a51254d464" containerID="f3cfe506f9dc3f4726beee9cad786b32451f4f13125ec40a63c1a6a0d6002ee8" exitCode=0
Jul 28 18:31:49.376772 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.376754    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"
Jul 28 18:31:49.376772 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.376756    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt" event={"ID":"2b7dad16-d1eb-463b-abf8-f5a51254d464","Type":"ContainerDied","Data":"f3cfe506f9dc3f4726beee9cad786b32451f4f13125ec40a63c1a6a0d6002ee8"}
Jul 28 18:31:49.377014 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.376859    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt" event={"ID":"2b7dad16-d1eb-463b-abf8-f5a51254d464","Type":"ContainerDied","Data":"e61feb35e046e1ec7e13cefcaedd45773f1bb3430e9309d48e5ef9597522ea38"}
Jul 28 18:31:49.377014 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.376899    2581 scope.go:117] "RemoveContainer" containerID="166c4e0f735156c078f79ac2559b691d607ea6dd98be8b555a34f363f12a53c9"
Jul 28 18:31:49.385828 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.385809    2581 scope.go:117] "RemoveContainer" containerID="f3cfe506f9dc3f4726beee9cad786b32451f4f13125ec40a63c1a6a0d6002ee8"
Jul 28 18:31:49.393322 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.393302    2581 scope.go:117] "RemoveContainer" containerID="55b738cfb1125d6ee2f0554130dc889af18bdce8269ab4df50ba896cc0b9ae30"
Jul 28 18:31:49.399633 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.399609    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"]
Jul 28 18:31:49.400747 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.400731    2581 scope.go:117] "RemoveContainer" containerID="166c4e0f735156c078f79ac2559b691d607ea6dd98be8b555a34f363f12a53c9"
Jul 28 18:31:49.401034 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:31:49.401014    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"166c4e0f735156c078f79ac2559b691d607ea6dd98be8b555a34f363f12a53c9\": container with ID starting with 166c4e0f735156c078f79ac2559b691d607ea6dd98be8b555a34f363f12a53c9 not found: ID does not exist" containerID="166c4e0f735156c078f79ac2559b691d607ea6dd98be8b555a34f363f12a53c9"
Jul 28 18:31:49.401093 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.401042    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"166c4e0f735156c078f79ac2559b691d607ea6dd98be8b555a34f363f12a53c9"} err="failed to get container status \"166c4e0f735156c078f79ac2559b691d607ea6dd98be8b555a34f363f12a53c9\": rpc error: code = NotFound desc = could not find container \"166c4e0f735156c078f79ac2559b691d607ea6dd98be8b555a34f363f12a53c9\": container with ID starting with 166c4e0f735156c078f79ac2559b691d607ea6dd98be8b555a34f363f12a53c9 not found: ID does not exist"
Jul 28 18:31:49.401093 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.401062    2581 scope.go:117] "RemoveContainer" containerID="f3cfe506f9dc3f4726beee9cad786b32451f4f13125ec40a63c1a6a0d6002ee8"
Jul 28 18:31:49.401424 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:31:49.401395    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3cfe506f9dc3f4726beee9cad786b32451f4f13125ec40a63c1a6a0d6002ee8\": container with ID starting with f3cfe506f9dc3f4726beee9cad786b32451f4f13125ec40a63c1a6a0d6002ee8 not found: ID does not exist" containerID="f3cfe506f9dc3f4726beee9cad786b32451f4f13125ec40a63c1a6a0d6002ee8"
Jul 28 18:31:49.401523 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.401428    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3cfe506f9dc3f4726beee9cad786b32451f4f13125ec40a63c1a6a0d6002ee8"} err="failed to get container status \"f3cfe506f9dc3f4726beee9cad786b32451f4f13125ec40a63c1a6a0d6002ee8\": rpc error: code = NotFound desc = could not find container \"f3cfe506f9dc3f4726beee9cad786b32451f4f13125ec40a63c1a6a0d6002ee8\": container with ID starting with f3cfe506f9dc3f4726beee9cad786b32451f4f13125ec40a63c1a6a0d6002ee8 not found: ID does not exist"
Jul 28 18:31:49.401523 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.401448    2581 scope.go:117] "RemoveContainer" containerID="55b738cfb1125d6ee2f0554130dc889af18bdce8269ab4df50ba896cc0b9ae30"
Jul 28 18:31:49.401729 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:31:49.401703    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55b738cfb1125d6ee2f0554130dc889af18bdce8269ab4df50ba896cc0b9ae30\": container with ID starting with 55b738cfb1125d6ee2f0554130dc889af18bdce8269ab4df50ba896cc0b9ae30 not found: ID does not exist" containerID="55b738cfb1125d6ee2f0554130dc889af18bdce8269ab4df50ba896cc0b9ae30"
Jul 28 18:31:49.401837 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.401732    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b738cfb1125d6ee2f0554130dc889af18bdce8269ab4df50ba896cc0b9ae30"} err="failed to get container status \"55b738cfb1125d6ee2f0554130dc889af18bdce8269ab4df50ba896cc0b9ae30\": rpc error: code = NotFound desc = could not find container \"55b738cfb1125d6ee2f0554130dc889af18bdce8269ab4df50ba896cc0b9ae30\": container with ID starting with 55b738cfb1125d6ee2f0554130dc889af18bdce8269ab4df50ba896cc0b9ae30 not found: ID does not exist"
Jul 28 18:31:49.403505 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:49.403483    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2w7rt"]
Jul 28 18:31:50.290610 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.290567    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b7dad16-d1eb-463b-abf8-f5a51254d464" path="/var/lib/kubelet/pods/2b7dad16-d1eb-463b-abf8-f5a51254d464/volumes"
Jul 28 18:31:50.408073 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.408027    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:31:50.449381 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.449341    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"]
Jul 28 18:31:50.449699 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.449685    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b7dad16-d1eb-463b-abf8-f5a51254d464" containerName="kserve-container"
Jul 28 18:31:50.449743 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.449702    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b7dad16-d1eb-463b-abf8-f5a51254d464" containerName="kserve-container"
Jul 28 18:31:50.449743 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.449725    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b7dad16-d1eb-463b-abf8-f5a51254d464" containerName="storage-initializer"
Jul 28 18:31:50.449743 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.449735    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b7dad16-d1eb-463b-abf8-f5a51254d464" containerName="storage-initializer"
Jul 28 18:31:50.449842 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.449747    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b7dad16-d1eb-463b-abf8-f5a51254d464" containerName="kube-rbac-proxy"
Jul 28 18:31:50.449842 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.449755    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b7dad16-d1eb-463b-abf8-f5a51254d464" containerName="kube-rbac-proxy"
Jul 28 18:31:50.449842 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.449774    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0863d75-fa9f-48e0-8a91-58928d288d2c" containerName="storage-initializer"
Jul 28 18:31:50.449842 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.449782    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0863d75-fa9f-48e0-8a91-58928d288d2c" containerName="storage-initializer"
Jul 28 18:31:50.449842 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.449793    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0863d75-fa9f-48e0-8a91-58928d288d2c" containerName="kube-rbac-proxy"
Jul 28 18:31:50.449842 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.449802    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0863d75-fa9f-48e0-8a91-58928d288d2c" containerName="kube-rbac-proxy"
Jul 28 18:31:50.449842 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.449812    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0863d75-fa9f-48e0-8a91-58928d288d2c" containerName="kserve-container"
Jul 28 18:31:50.449842 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.449820    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0863d75-fa9f-48e0-8a91-58928d288d2c" containerName="kserve-container"
Jul 28 18:31:50.450142 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.449876    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0863d75-fa9f-48e0-8a91-58928d288d2c" containerName="kserve-container"
Jul 28 18:31:50.450142 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.449888    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b7dad16-d1eb-463b-abf8-f5a51254d464" containerName="kube-rbac-proxy"
Jul 28 18:31:50.450142 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.449898    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b7dad16-d1eb-463b-abf8-f5a51254d464" containerName="kserve-container"
Jul 28 18:31:50.450142 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.449907    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0863d75-fa9f-48e0-8a91-58928d288d2c" containerName="kube-rbac-proxy"
Jul 28 18:31:50.453399 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.453375    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"
Jul 28 18:31:50.456122 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.456083    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Jul 28 18:31:50.456122 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.456117    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 18:31:50.456305 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.456179    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 18:31:50.456408 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.456390    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:31:50.457420 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.457401    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:31:50.457532 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.457467    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:31:50.457532 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.457516    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:31:50.463291 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.463270    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"]
Jul 28 18:31:50.527742 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.527696    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr\" (UID: \"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"
Jul 28 18:31:50.527742 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.527741    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr\" (UID: \"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"
Jul 28 18:31:50.528003 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.527767    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr\" (UID: \"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"
Jul 28 18:31:50.528003 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.527825    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx9f5\" (UniqueName: \"kubernetes.io/projected/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-kube-api-access-wx9f5\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr\" (UID: \"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"
Jul 28 18:31:50.528003 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.527879    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr\" (UID: \"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"
Jul 28 18:31:50.628341 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.628255    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx9f5\" (UniqueName: \"kubernetes.io/projected/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-kube-api-access-wx9f5\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr\" (UID: \"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"
Jul 28 18:31:50.628341 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.628309    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr\" (UID: \"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"
Jul 28 18:31:50.628517 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.628363    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr\" (UID: \"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"
Jul 28 18:31:50.628517 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.628392    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr\" (UID: \"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"
Jul 28 18:31:50.628630 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.628541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr\" (UID: \"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"
Jul 28 18:31:50.628701 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:31:50.628680    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-serving-cert: secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Jul 28 18:31:50.628786 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:31:50.628773    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-proxy-tls podName:ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1 nodeName:}" failed. No retries permitted until 2026-07-28 18:31:51.128753743 +0000 UTC m=+4959.403370169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-proxy-tls") pod "isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr" (UID: "ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1") : secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Jul 28 18:31:50.628786 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.628775    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr\" (UID: \"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"
Jul 28 18:31:50.629104 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.629085    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr\" (UID: \"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"
Jul 28 18:31:50.629170 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.629124    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr\" (UID: \"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"
Jul 28 18:31:50.637084 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:50.637064    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx9f5\" (UniqueName: \"kubernetes.io/projected/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-kube-api-access-wx9f5\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr\" (UID: \"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"
Jul 28 18:31:51.094998 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:51.094964    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:31:51.131259 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:51.131224    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr\" (UID: \"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"
Jul 28 18:31:51.133795 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:51.133773    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr\" (UID: \"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"
Jul 28 18:31:51.363960 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:51.363836    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"
Jul 28 18:31:51.494381 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:51.494348    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"]
Jul 28 18:31:51.498315 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:31:51.498288    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca41fd3b_8466_4e52_b9f9_c3e8f25c2da1.slice/crio-8190e0b7555ad5c964ed6ac197bf96a6b166989f68cf9f12de573d0965519dce WatchSource:0}: Error finding container 8190e0b7555ad5c964ed6ac197bf96a6b166989f68cf9f12de573d0965519dce: Status 404 returned error can't find the container with id 8190e0b7555ad5c964ed6ac197bf96a6b166989f68cf9f12de573d0965519dce
Jul 28 18:31:52.391789 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:52.391749    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr" event={"ID":"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1","Type":"ContainerStarted","Data":"5797a4188a50fcf4d85bccdb17fb412dcc8274d8cf7bb395dc5b755d9149af8c"}
Jul 28 18:31:52.392209 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:52.391811    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr" event={"ID":"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1","Type":"ContainerStarted","Data":"8190e0b7555ad5c964ed6ac197bf96a6b166989f68cf9f12de573d0965519dce"}
Jul 28 18:31:55.404325 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:55.404287    2581 generic.go:356] "Generic (PLEG): container finished" podID="ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1" containerID="5797a4188a50fcf4d85bccdb17fb412dcc8274d8cf7bb395dc5b755d9149af8c" exitCode=0
Jul 28 18:31:55.404732 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:55.404331    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr" event={"ID":"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1","Type":"ContainerDied","Data":"5797a4188a50fcf4d85bccdb17fb412dcc8274d8cf7bb395dc5b755d9149af8c"}
Jul 28 18:31:56.409396 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:56.409359    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr" event={"ID":"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1","Type":"ContainerStarted","Data":"eecc3a05a44128fc428d8c038f78e98a1be9111cfae11135827af3ed6bd7427f"}
Jul 28 18:31:56.409813 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:56.409419    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr" event={"ID":"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1","Type":"ContainerStarted","Data":"73d70fe78aa917eeb30888d7da0cf003e9cfaa5d3cd368ed44afa2c78ca534fd"}
Jul 28 18:31:56.409813 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:56.409729    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"
Jul 28 18:31:56.409813 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:56.409794    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"
Jul 28 18:31:56.430388 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:31:56.430334    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr" podStartSLOduration=2.770868284 podStartE2EDuration="6.430316343s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.659448059s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:31:56.428735257 +0000 UTC m=+4964.703351691" watchObservedRunningTime="2026-07-28 18:31:56.430316343 +0000 UTC m=+4964.704932774"
Jul 28 18:32:02.418285 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:02.418253    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"
Jul 28 18:32:32.658544 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:32.658497    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr" podUID="ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 28 18:32:42.421005 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:42.420934    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"
Jul 28 18:32:50.556687 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:50.556644    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"]
Jul 28 18:32:50.557130 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:50.556933    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr" podUID="ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1" containerName="kserve-container" containerID="cri-o://73d70fe78aa917eeb30888d7da0cf003e9cfaa5d3cd368ed44afa2c78ca534fd" gracePeriod=30
Jul 28 18:32:50.557130 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:50.556999    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr" podUID="ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1" containerName="kube-rbac-proxy" containerID="cri-o://eecc3a05a44128fc428d8c038f78e98a1be9111cfae11135827af3ed6bd7427f" gracePeriod=30
Jul 28 18:32:50.656092 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:50.656052    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:32:51.598299 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:51.598264    2581 generic.go:356] "Generic (PLEG): container finished" podID="ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1" containerID="eecc3a05a44128fc428d8c038f78e98a1be9111cfae11135827af3ed6bd7427f" exitCode=2
Jul 28 18:32:51.598672 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:51.598324    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr" event={"ID":"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1","Type":"ContainerDied","Data":"eecc3a05a44128fc428d8c038f78e98a1be9111cfae11135827af3ed6bd7427f"}
Jul 28 18:32:52.413743 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:52.413692    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr" podUID="ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.67:8643/healthz\": dial tcp 10.133.0.67:8643: connect: connection refused"
Jul 28 18:32:53.460189 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:53.460126    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr" podUID="ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.67:8080/v2/models/isvc-xgboost-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 18:32:55.666389 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:55.666358    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:32:55.698921 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:55.698879    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"]
Jul 28 18:32:55.702348 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:55.702324    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"
Jul 28 18:32:55.704862 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:55.704831    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 28 18:32:55.705016 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:55.704883    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Jul 28 18:32:55.713274 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:55.713252    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"]
Jul 28 18:32:55.829808 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:55.829766    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22231a65-b15d-4d65-b8bb-b0a93f23344e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bpptz\" (UID: \"22231a65-b15d-4d65-b8bb-b0a93f23344e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"
Jul 28 18:32:55.830047 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:55.829848    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22231a65-b15d-4d65-b8bb-b0a93f23344e-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bpptz\" (UID: \"22231a65-b15d-4d65-b8bb-b0a93f23344e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"
Jul 28 18:32:55.830047 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:55.829888    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22231a65-b15d-4d65-b8bb-b0a93f23344e-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bpptz\" (UID: \"22231a65-b15d-4d65-b8bb-b0a93f23344e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"
Jul 28 18:32:55.830047 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:55.829914    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22231a65-b15d-4d65-b8bb-b0a93f23344e-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bpptz\" (UID: \"22231a65-b15d-4d65-b8bb-b0a93f23344e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"
Jul 28 18:32:55.830047 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:55.830015    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgsv9\" (UniqueName: \"kubernetes.io/projected/22231a65-b15d-4d65-b8bb-b0a93f23344e-kube-api-access-wgsv9\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bpptz\" (UID: \"22231a65-b15d-4d65-b8bb-b0a93f23344e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"
Jul 28 18:32:55.931224 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:55.931131    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22231a65-b15d-4d65-b8bb-b0a93f23344e-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bpptz\" (UID: \"22231a65-b15d-4d65-b8bb-b0a93f23344e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"
Jul 28 18:32:55.931224 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:55.931185    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22231a65-b15d-4d65-b8bb-b0a93f23344e-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bpptz\" (UID: \"22231a65-b15d-4d65-b8bb-b0a93f23344e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"
Jul 28 18:32:55.931224 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:55.931206    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22231a65-b15d-4d65-b8bb-b0a93f23344e-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bpptz\" (UID: \"22231a65-b15d-4d65-b8bb-b0a93f23344e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"
Jul 28 18:32:55.931224 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:55.931229    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgsv9\" (UniqueName: \"kubernetes.io/projected/22231a65-b15d-4d65-b8bb-b0a93f23344e-kube-api-access-wgsv9\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bpptz\" (UID: \"22231a65-b15d-4d65-b8bb-b0a93f23344e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"
Jul 28 18:32:55.931539 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:55.931256    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22231a65-b15d-4d65-b8bb-b0a93f23344e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bpptz\" (UID: \"22231a65-b15d-4d65-b8bb-b0a93f23344e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"
Jul 28 18:32:55.931539 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:32:55.931326    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
Jul 28 18:32:55.931539 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:32:55.931407    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22231a65-b15d-4d65-b8bb-b0a93f23344e-proxy-tls podName:22231a65-b15d-4d65-b8bb-b0a93f23344e nodeName:}" failed. No retries permitted until 2026-07-28 18:32:56.431385182 +0000 UTC m=+5024.706001599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/22231a65-b15d-4d65-b8bb-b0a93f23344e-proxy-tls") pod "isvc-xgboost-v2-predictor-fcc668bc-bpptz" (UID: "22231a65-b15d-4d65-b8bb-b0a93f23344e") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
Jul 28 18:32:55.931668 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:55.931600    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22231a65-b15d-4d65-b8bb-b0a93f23344e-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bpptz\" (UID: \"22231a65-b15d-4d65-b8bb-b0a93f23344e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"
Jul 28 18:32:55.932077 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:55.932057    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22231a65-b15d-4d65-b8bb-b0a93f23344e-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bpptz\" (UID: \"22231a65-b15d-4d65-b8bb-b0a93f23344e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"
Jul 28 18:32:55.932119 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:55.932057    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22231a65-b15d-4d65-b8bb-b0a93f23344e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bpptz\" (UID: \"22231a65-b15d-4d65-b8bb-b0a93f23344e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"
Jul 28 18:32:55.939747 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:55.939722    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgsv9\" (UniqueName: \"kubernetes.io/projected/22231a65-b15d-4d65-b8bb-b0a93f23344e-kube-api-access-wgsv9\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bpptz\" (UID: \"22231a65-b15d-4d65-b8bb-b0a93f23344e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"
Jul 28 18:32:56.435354 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:56.435322    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22231a65-b15d-4d65-b8bb-b0a93f23344e-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bpptz\" (UID: \"22231a65-b15d-4d65-b8bb-b0a93f23344e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"
Jul 28 18:32:56.435572 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:32:56.435511    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
Jul 28 18:32:56.435648 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:32:56.435592    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22231a65-b15d-4d65-b8bb-b0a93f23344e-proxy-tls podName:22231a65-b15d-4d65-b8bb-b0a93f23344e nodeName:}" failed. No retries permitted until 2026-07-28 18:32:57.435567964 +0000 UTC m=+5025.710184373 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/22231a65-b15d-4d65-b8bb-b0a93f23344e-proxy-tls") pod "isvc-xgboost-v2-predictor-fcc668bc-bpptz" (UID: "22231a65-b15d-4d65-b8bb-b0a93f23344e") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
Jul 28 18:32:56.542787 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:56.542745    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:32:57.413875 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:57.413830    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr" podUID="ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.67:8643/healthz\": dial tcp 10.133.0.67:8643: connect: connection refused"
Jul 28 18:32:57.443543 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:57.443510    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22231a65-b15d-4d65-b8bb-b0a93f23344e-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bpptz\" (UID: \"22231a65-b15d-4d65-b8bb-b0a93f23344e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"
Jul 28 18:32:57.445972 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:57.445934    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22231a65-b15d-4d65-b8bb-b0a93f23344e-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-bpptz\" (UID: \"22231a65-b15d-4d65-b8bb-b0a93f23344e\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"
Jul 28 18:32:57.512044 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:57.512002    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"
Jul 28 18:32:57.651065 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:57.651043    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"]
Jul 28 18:32:57.653216 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:32:57.653190    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22231a65_b15d_4d65_b8bb_b0a93f23344e.slice/crio-4cd1b0ade70f0956acd9de1e27184cc23c2fc6de45db020f3539dcf01ebb4afc WatchSource:0}: Error finding container 4cd1b0ade70f0956acd9de1e27184cc23c2fc6de45db020f3539dcf01ebb4afc: Status 404 returned error can't find the container with id 4cd1b0ade70f0956acd9de1e27184cc23c2fc6de45db020f3539dcf01ebb4afc
Jul 28 18:32:58.501804 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.501777    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"
Jul 28 18:32:58.624291 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.624198    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz" event={"ID":"22231a65-b15d-4d65-b8bb-b0a93f23344e","Type":"ContainerStarted","Data":"453690544f287a26be6a07f81d490736e65c8f23a61e69225083cb98d5e11deb"}
Jul 28 18:32:58.624291 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.624269    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz" event={"ID":"22231a65-b15d-4d65-b8bb-b0a93f23344e","Type":"ContainerStarted","Data":"4cd1b0ade70f0956acd9de1e27184cc23c2fc6de45db020f3539dcf01ebb4afc"}
Jul 28 18:32:58.625974 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.625935    2581 generic.go:356] "Generic (PLEG): container finished" podID="ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1" containerID="73d70fe78aa917eeb30888d7da0cf003e9cfaa5d3cd368ed44afa2c78ca534fd" exitCode=0
Jul 28 18:32:58.626091 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.626000    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"
Jul 28 18:32:58.626091 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.626008    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr" event={"ID":"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1","Type":"ContainerDied","Data":"73d70fe78aa917eeb30888d7da0cf003e9cfaa5d3cd368ed44afa2c78ca534fd"}
Jul 28 18:32:58.626091 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.626053    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr" event={"ID":"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1","Type":"ContainerDied","Data":"8190e0b7555ad5c964ed6ac197bf96a6b166989f68cf9f12de573d0965519dce"}
Jul 28 18:32:58.626091 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.626081    2581 scope.go:117] "RemoveContainer" containerID="eecc3a05a44128fc428d8c038f78e98a1be9111cfae11135827af3ed6bd7427f"
Jul 28 18:32:58.633647 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.633629    2581 scope.go:117] "RemoveContainer" containerID="73d70fe78aa917eeb30888d7da0cf003e9cfaa5d3cd368ed44afa2c78ca534fd"
Jul 28 18:32:58.640565 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.640543    2581 scope.go:117] "RemoveContainer" containerID="5797a4188a50fcf4d85bccdb17fb412dcc8274d8cf7bb395dc5b755d9149af8c"
Jul 28 18:32:58.648095 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.648077    2581 scope.go:117] "RemoveContainer" containerID="eecc3a05a44128fc428d8c038f78e98a1be9111cfae11135827af3ed6bd7427f"
Jul 28 18:32:58.648352 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:32:58.648332    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eecc3a05a44128fc428d8c038f78e98a1be9111cfae11135827af3ed6bd7427f\": container with ID starting with eecc3a05a44128fc428d8c038f78e98a1be9111cfae11135827af3ed6bd7427f not found: ID does not exist" containerID="eecc3a05a44128fc428d8c038f78e98a1be9111cfae11135827af3ed6bd7427f"
Jul 28 18:32:58.648409 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.648361    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eecc3a05a44128fc428d8c038f78e98a1be9111cfae11135827af3ed6bd7427f"} err="failed to get container status \"eecc3a05a44128fc428d8c038f78e98a1be9111cfae11135827af3ed6bd7427f\": rpc error: code = NotFound desc = could not find container \"eecc3a05a44128fc428d8c038f78e98a1be9111cfae11135827af3ed6bd7427f\": container with ID starting with eecc3a05a44128fc428d8c038f78e98a1be9111cfae11135827af3ed6bd7427f not found: ID does not exist"
Jul 28 18:32:58.648409 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.648387    2581 scope.go:117] "RemoveContainer" containerID="73d70fe78aa917eeb30888d7da0cf003e9cfaa5d3cd368ed44afa2c78ca534fd"
Jul 28 18:32:58.648591 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:32:58.648577    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d70fe78aa917eeb30888d7da0cf003e9cfaa5d3cd368ed44afa2c78ca534fd\": container with ID starting with 73d70fe78aa917eeb30888d7da0cf003e9cfaa5d3cd368ed44afa2c78ca534fd not found: ID does not exist" containerID="73d70fe78aa917eeb30888d7da0cf003e9cfaa5d3cd368ed44afa2c78ca534fd"
Jul 28 18:32:58.648632 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.648595    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d70fe78aa917eeb30888d7da0cf003e9cfaa5d3cd368ed44afa2c78ca534fd"} err="failed to get container status \"73d70fe78aa917eeb30888d7da0cf003e9cfaa5d3cd368ed44afa2c78ca534fd\": rpc error: code = NotFound desc = could not find container \"73d70fe78aa917eeb30888d7da0cf003e9cfaa5d3cd368ed44afa2c78ca534fd\": container with ID starting with 73d70fe78aa917eeb30888d7da0cf003e9cfaa5d3cd368ed44afa2c78ca534fd not found: ID does not exist"
Jul 28 18:32:58.648632 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.648610    2581 scope.go:117] "RemoveContainer" containerID="5797a4188a50fcf4d85bccdb17fb412dcc8274d8cf7bb395dc5b755d9149af8c"
Jul 28 18:32:58.648788 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:32:58.648772    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5797a4188a50fcf4d85bccdb17fb412dcc8274d8cf7bb395dc5b755d9149af8c\": container with ID starting with 5797a4188a50fcf4d85bccdb17fb412dcc8274d8cf7bb395dc5b755d9149af8c not found: ID does not exist" containerID="5797a4188a50fcf4d85bccdb17fb412dcc8274d8cf7bb395dc5b755d9149af8c"
Jul 28 18:32:58.648824 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.648794    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5797a4188a50fcf4d85bccdb17fb412dcc8274d8cf7bb395dc5b755d9149af8c"} err="failed to get container status \"5797a4188a50fcf4d85bccdb17fb412dcc8274d8cf7bb395dc5b755d9149af8c\": rpc error: code = NotFound desc = could not find container \"5797a4188a50fcf4d85bccdb17fb412dcc8274d8cf7bb395dc5b755d9149af8c\": container with ID starting with 5797a4188a50fcf4d85bccdb17fb412dcc8274d8cf7bb395dc5b755d9149af8c not found: ID does not exist"
Jul 28 18:32:58.652469 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.652450    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-kube-rbac-proxy-sar-config\") pod \"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1\" (UID: \"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1\") "
Jul 28 18:32:58.652561 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.652527    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-cabundle-cert\") pod \"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1\" (UID: \"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1\") "
Jul 28 18:32:58.652625 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.652563    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wx9f5\" (UniqueName: \"kubernetes.io/projected/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-kube-api-access-wx9f5\") pod \"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1\" (UID: \"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1\") "
Jul 28 18:32:58.652625 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.652598    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-kserve-provision-location\") pod \"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1\" (UID: \"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1\") "
Jul 28 18:32:58.652725 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.652649    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-proxy-tls\") pod \"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1\" (UID: \"ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1\") "
Jul 28 18:32:58.652909 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.652885    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1" (UID: "ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:32:58.652909 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.652894    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1" (UID: "ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:32:58.653069 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.652925    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1" (UID: "ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:32:58.654641 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.654609    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-kube-api-access-wx9f5" (OuterVolumeSpecName: "kube-api-access-wx9f5") pod "ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1" (UID: "ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1"). InnerVolumeSpecName "kube-api-access-wx9f5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:32:58.654641 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.654624    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1" (UID: "ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:32:58.753496 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.753460    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:32:58.753496 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.753488    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx9f5\" (UniqueName: \"kubernetes.io/projected/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-kube-api-access-wx9f5\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:32:58.753496 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.753502    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:32:58.753767 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.753515    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:32:58.753767 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.753527    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:32:58.949148 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.949116    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"]
Jul 28 18:32:58.954523 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:32:58.954494    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v72gr"]
Jul 28 18:33:00.289628 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:33:00.289589    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1" path="/var/lib/kubelet/pods/ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1/volumes"
Jul 28 18:33:01.639364 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:33:01.639327    2581 generic.go:356] "Generic (PLEG): container finished" podID="22231a65-b15d-4d65-b8bb-b0a93f23344e" containerID="453690544f287a26be6a07f81d490736e65c8f23a61e69225083cb98d5e11deb" exitCode=0
Jul 28 18:33:01.639765 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:33:01.639408    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz" event={"ID":"22231a65-b15d-4d65-b8bb-b0a93f23344e","Type":"ContainerDied","Data":"453690544f287a26be6a07f81d490736e65c8f23a61e69225083cb98d5e11deb"}
Jul 28 18:33:02.644727 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:33:02.644693    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz" event={"ID":"22231a65-b15d-4d65-b8bb-b0a93f23344e","Type":"ContainerStarted","Data":"45f00a3269a1bbca0348850f1bb72adc2de420dad6b1d4835635bb7121b5d3b6"}
Jul 28 18:33:02.645172 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:33:02.644756    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz" event={"ID":"22231a65-b15d-4d65-b8bb-b0a93f23344e","Type":"ContainerStarted","Data":"812ca62fbfdcdd904d47cad7d87637dee792b86ecf4ccb9c9bc1a5ad68db1ef8"}
Jul 28 18:33:02.645172 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:33:02.644972    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"
Jul 28 18:33:02.645172 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:33:02.645030    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"
Jul 28 18:33:02.646440 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:33:02.646412    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz" podUID="22231a65-b15d-4d65-b8bb-b0a93f23344e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Jul 28 18:33:02.666305 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:33:02.666251    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz" podStartSLOduration=3.936224064 podStartE2EDuration="7.666237425s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.730013361s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:33:02.664451616 +0000 UTC m=+5030.939068049" watchObservedRunningTime="2026-07-28 18:33:02.666237425 +0000 UTC m=+5030.940853855"
Jul 28 18:33:03.648180 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:33:03.648134    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz" podUID="22231a65-b15d-4d65-b8bb-b0a93f23344e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Jul 28 18:33:08.653382 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:33:08.653299    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"
Jul 28 18:33:08.653910 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:33:08.653880    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz" podUID="22231a65-b15d-4d65-b8bb-b0a93f23344e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Jul 28 18:33:18.653787 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:33:18.653747    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz" podUID="22231a65-b15d-4d65-b8bb-b0a93f23344e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Jul 28 18:33:28.654159 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:33:28.654115    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz" podUID="22231a65-b15d-4d65-b8bb-b0a93f23344e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Jul 28 18:33:38.654324 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:33:38.654283    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz" podUID="22231a65-b15d-4d65-b8bb-b0a93f23344e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Jul 28 18:33:48.654421 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:33:48.654377    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz" podUID="22231a65-b15d-4d65-b8bb-b0a93f23344e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Jul 28 18:33:58.654129 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:33:58.654088    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz" podUID="22231a65-b15d-4d65-b8bb-b0a93f23344e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Jul 28 18:34:08.654696 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:08.654664    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"
Jul 28 18:34:12.506624 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:12.506588    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 18:34:12.518158 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:12.518133    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 18:34:15.815546 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:15.815508    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"]
Jul 28 18:34:15.815969 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:15.815785    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz" podUID="22231a65-b15d-4d65-b8bb-b0a93f23344e" containerName="kserve-container" containerID="cri-o://812ca62fbfdcdd904d47cad7d87637dee792b86ecf4ccb9c9bc1a5ad68db1ef8" gracePeriod=30
Jul 28 18:34:15.815969 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:15.815863    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz" podUID="22231a65-b15d-4d65-b8bb-b0a93f23344e" containerName="kube-rbac-proxy" containerID="cri-o://45f00a3269a1bbca0348850f1bb72adc2de420dad6b1d4835635bb7121b5d3b6" gracePeriod=30
Jul 28 18:34:16.643823 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:16.643791    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:34:16.905213 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:16.905178    2581 generic.go:356] "Generic (PLEG): container finished" podID="22231a65-b15d-4d65-b8bb-b0a93f23344e" containerID="45f00a3269a1bbca0348850f1bb72adc2de420dad6b1d4835635bb7121b5d3b6" exitCode=2
Jul 28 18:34:16.905602 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:16.905249    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz" event={"ID":"22231a65-b15d-4d65-b8bb-b0a93f23344e","Type":"ContainerDied","Data":"45f00a3269a1bbca0348850f1bb72adc2de420dad6b1d4835635bb7121b5d3b6"}
Jul 28 18:34:18.648539 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:18.648489    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz" podUID="22231a65-b15d-4d65-b8bb-b0a93f23344e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.68:8643/healthz\": dial tcp 10.133.0.68:8643: connect: connection refused"
Jul 28 18:34:18.653907 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:18.653878    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz" podUID="22231a65-b15d-4d65-b8bb-b0a93f23344e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.68:8080: connect: connection refused"
Jul 28 18:34:19.662861 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.662830    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"
Jul 28 18:34:19.763588 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.763488    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22231a65-b15d-4d65-b8bb-b0a93f23344e-kube-rbac-proxy-sar-config\") pod \"22231a65-b15d-4d65-b8bb-b0a93f23344e\" (UID: \"22231a65-b15d-4d65-b8bb-b0a93f23344e\") "
Jul 28 18:34:19.763588 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.763535    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgsv9\" (UniqueName: \"kubernetes.io/projected/22231a65-b15d-4d65-b8bb-b0a93f23344e-kube-api-access-wgsv9\") pod \"22231a65-b15d-4d65-b8bb-b0a93f23344e\" (UID: \"22231a65-b15d-4d65-b8bb-b0a93f23344e\") "
Jul 28 18:34:19.763588 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.763582    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22231a65-b15d-4d65-b8bb-b0a93f23344e-proxy-tls\") pod \"22231a65-b15d-4d65-b8bb-b0a93f23344e\" (UID: \"22231a65-b15d-4d65-b8bb-b0a93f23344e\") "
Jul 28 18:34:19.763901 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.763603    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22231a65-b15d-4d65-b8bb-b0a93f23344e-cabundle-cert\") pod \"22231a65-b15d-4d65-b8bb-b0a93f23344e\" (UID: \"22231a65-b15d-4d65-b8bb-b0a93f23344e\") "
Jul 28 18:34:19.763901 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.763634    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22231a65-b15d-4d65-b8bb-b0a93f23344e-kserve-provision-location\") pod \"22231a65-b15d-4d65-b8bb-b0a93f23344e\" (UID: \"22231a65-b15d-4d65-b8bb-b0a93f23344e\") "
Jul 28 18:34:19.764037 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.763971    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22231a65-b15d-4d65-b8bb-b0a93f23344e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "22231a65-b15d-4d65-b8bb-b0a93f23344e" (UID: "22231a65-b15d-4d65-b8bb-b0a93f23344e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:34:19.764037 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.764014    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22231a65-b15d-4d65-b8bb-b0a93f23344e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "22231a65-b15d-4d65-b8bb-b0a93f23344e" (UID: "22231a65-b15d-4d65-b8bb-b0a93f23344e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:34:19.764114 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.764038    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22231a65-b15d-4d65-b8bb-b0a93f23344e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "22231a65-b15d-4d65-b8bb-b0a93f23344e" (UID: "22231a65-b15d-4d65-b8bb-b0a93f23344e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:34:19.765738 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.765714    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22231a65-b15d-4d65-b8bb-b0a93f23344e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "22231a65-b15d-4d65-b8bb-b0a93f23344e" (UID: "22231a65-b15d-4d65-b8bb-b0a93f23344e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:34:19.765846 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.765772    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22231a65-b15d-4d65-b8bb-b0a93f23344e-kube-api-access-wgsv9" (OuterVolumeSpecName: "kube-api-access-wgsv9") pod "22231a65-b15d-4d65-b8bb-b0a93f23344e" (UID: "22231a65-b15d-4d65-b8bb-b0a93f23344e"). InnerVolumeSpecName "kube-api-access-wgsv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:34:19.864463 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.864424    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22231a65-b15d-4d65-b8bb-b0a93f23344e-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:34:19.864463 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.864455    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22231a65-b15d-4d65-b8bb-b0a93f23344e-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:34:19.864463 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.864465    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22231a65-b15d-4d65-b8bb-b0a93f23344e-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:34:19.864704 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.864473    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22231a65-b15d-4d65-b8bb-b0a93f23344e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:34:19.864704 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.864484    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgsv9\" (UniqueName: \"kubernetes.io/projected/22231a65-b15d-4d65-b8bb-b0a93f23344e-kube-api-access-wgsv9\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:34:19.916801 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.916765    2581 generic.go:356] "Generic (PLEG): container finished" podID="22231a65-b15d-4d65-b8bb-b0a93f23344e" containerID="812ca62fbfdcdd904d47cad7d87637dee792b86ecf4ccb9c9bc1a5ad68db1ef8" exitCode=0
Jul 28 18:34:19.916986 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.916804    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz" event={"ID":"22231a65-b15d-4d65-b8bb-b0a93f23344e","Type":"ContainerDied","Data":"812ca62fbfdcdd904d47cad7d87637dee792b86ecf4ccb9c9bc1a5ad68db1ef8"}
Jul 28 18:34:19.916986 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.916823    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"
Jul 28 18:34:19.916986 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.916873    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz" event={"ID":"22231a65-b15d-4d65-b8bb-b0a93f23344e","Type":"ContainerDied","Data":"4cd1b0ade70f0956acd9de1e27184cc23c2fc6de45db020f3539dcf01ebb4afc"}
Jul 28 18:34:19.916986 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.916901    2581 scope.go:117] "RemoveContainer" containerID="45f00a3269a1bbca0348850f1bb72adc2de420dad6b1d4835635bb7121b5d3b6"
Jul 28 18:34:19.926251 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.926214    2581 scope.go:117] "RemoveContainer" containerID="812ca62fbfdcdd904d47cad7d87637dee792b86ecf4ccb9c9bc1a5ad68db1ef8"
Jul 28 18:34:19.935525 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.935504    2581 scope.go:117] "RemoveContainer" containerID="453690544f287a26be6a07f81d490736e65c8f23a61e69225083cb98d5e11deb"
Jul 28 18:34:19.940337 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.940314    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"]
Jul 28 18:34:19.944607 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.944546    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-bpptz"]
Jul 28 18:34:19.944670 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.944648    2581 scope.go:117] "RemoveContainer" containerID="45f00a3269a1bbca0348850f1bb72adc2de420dad6b1d4835635bb7121b5d3b6"
Jul 28 18:34:19.944967 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:34:19.944923    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45f00a3269a1bbca0348850f1bb72adc2de420dad6b1d4835635bb7121b5d3b6\": container with ID starting with 45f00a3269a1bbca0348850f1bb72adc2de420dad6b1d4835635bb7121b5d3b6 not found: ID does not exist" containerID="45f00a3269a1bbca0348850f1bb72adc2de420dad6b1d4835635bb7121b5d3b6"
Jul 28 18:34:19.945071 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.944973    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45f00a3269a1bbca0348850f1bb72adc2de420dad6b1d4835635bb7121b5d3b6"} err="failed to get container status \"45f00a3269a1bbca0348850f1bb72adc2de420dad6b1d4835635bb7121b5d3b6\": rpc error: code = NotFound desc = could not find container \"45f00a3269a1bbca0348850f1bb72adc2de420dad6b1d4835635bb7121b5d3b6\": container with ID starting with 45f00a3269a1bbca0348850f1bb72adc2de420dad6b1d4835635bb7121b5d3b6 not found: ID does not exist"
Jul 28 18:34:19.945071 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.944993    2581 scope.go:117] "RemoveContainer" containerID="812ca62fbfdcdd904d47cad7d87637dee792b86ecf4ccb9c9bc1a5ad68db1ef8"
Jul 28 18:34:19.945261 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:34:19.945242    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"812ca62fbfdcdd904d47cad7d87637dee792b86ecf4ccb9c9bc1a5ad68db1ef8\": container with ID starting with 812ca62fbfdcdd904d47cad7d87637dee792b86ecf4ccb9c9bc1a5ad68db1ef8 not found: ID does not exist" containerID="812ca62fbfdcdd904d47cad7d87637dee792b86ecf4ccb9c9bc1a5ad68db1ef8"
Jul 28 18:34:19.945308 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.945268    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"812ca62fbfdcdd904d47cad7d87637dee792b86ecf4ccb9c9bc1a5ad68db1ef8"} err="failed to get container status \"812ca62fbfdcdd904d47cad7d87637dee792b86ecf4ccb9c9bc1a5ad68db1ef8\": rpc error: code = NotFound desc = could not find container \"812ca62fbfdcdd904d47cad7d87637dee792b86ecf4ccb9c9bc1a5ad68db1ef8\": container with ID starting with 812ca62fbfdcdd904d47cad7d87637dee792b86ecf4ccb9c9bc1a5ad68db1ef8 not found: ID does not exist"
Jul 28 18:34:19.945308 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.945284    2581 scope.go:117] "RemoveContainer" containerID="453690544f287a26be6a07f81d490736e65c8f23a61e69225083cb98d5e11deb"
Jul 28 18:34:19.945528 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:34:19.945512    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"453690544f287a26be6a07f81d490736e65c8f23a61e69225083cb98d5e11deb\": container with ID starting with 453690544f287a26be6a07f81d490736e65c8f23a61e69225083cb98d5e11deb not found: ID does not exist" containerID="453690544f287a26be6a07f81d490736e65c8f23a61e69225083cb98d5e11deb"
Jul 28 18:34:19.945571 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:19.945532    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"453690544f287a26be6a07f81d490736e65c8f23a61e69225083cb98d5e11deb"} err="failed to get container status \"453690544f287a26be6a07f81d490736e65c8f23a61e69225083cb98d5e11deb\": rpc error: code = NotFound desc = could not find container \"453690544f287a26be6a07f81d490736e65c8f23a61e69225083cb98d5e11deb\": container with ID starting with 453690544f287a26be6a07f81d490736e65c8f23a61e69225083cb98d5e11deb not found: ID does not exist"
Jul 28 18:34:20.290030 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:20.289998    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22231a65-b15d-4d65-b8bb-b0a93f23344e" path="/var/lib/kubelet/pods/22231a65-b15d-4d65-b8bb-b0a93f23344e/volumes"
Jul 28 18:34:20.950363 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:20.950323    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:34:20.976203 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:20.976167    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh"]
Jul 28 18:34:20.976425 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:20.976413    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22231a65-b15d-4d65-b8bb-b0a93f23344e" containerName="storage-initializer"
Jul 28 18:34:20.976492 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:20.976426    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="22231a65-b15d-4d65-b8bb-b0a93f23344e" containerName="storage-initializer"
Jul 28 18:34:20.976492 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:20.976440    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1" containerName="kserve-container"
Jul 28 18:34:20.976492 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:20.976446    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1" containerName="kserve-container"
Jul 28 18:34:20.976492 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:20.976453    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22231a65-b15d-4d65-b8bb-b0a93f23344e" containerName="kserve-container"
Jul 28 18:34:20.976492 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:20.976459    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="22231a65-b15d-4d65-b8bb-b0a93f23344e" containerName="kserve-container"
Jul 28 18:34:20.976492 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:20.976472    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1" containerName="storage-initializer"
Jul 28 18:34:20.976492 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:20.976477    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1" containerName="storage-initializer"
Jul 28 18:34:20.976492 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:20.976486    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22231a65-b15d-4d65-b8bb-b0a93f23344e" containerName="kube-rbac-proxy"
Jul 28 18:34:20.976492 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:20.976491    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="22231a65-b15d-4d65-b8bb-b0a93f23344e" containerName="kube-rbac-proxy"
Jul 28 18:34:20.976767 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:20.976498    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1" containerName="kube-rbac-proxy"
Jul 28 18:34:20.976767 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:20.976503    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1" containerName="kube-rbac-proxy"
Jul 28 18:34:20.976767 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:20.976539    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1" containerName="kserve-container"
Jul 28 18:34:20.976767 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:20.976547    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="22231a65-b15d-4d65-b8bb-b0a93f23344e" containerName="kserve-container"
Jul 28 18:34:20.976767 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:20.976554    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca41fd3b-8466-4e52-b9f9-c3e8f25c2da1" containerName="kube-rbac-proxy"
Jul 28 18:34:20.976767 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:20.976560    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="22231a65-b15d-4d65-b8bb-b0a93f23344e" containerName="kube-rbac-proxy"
Jul 28 18:34:20.979468 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:20.979446    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh"
Jul 28 18:34:20.982155 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:20.982130    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:34:20.982155 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:20.982153    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 28 18:34:20.982349 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:20.982284    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:34:20.982406 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:20.982361    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Jul 28 18:34:20.983365 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:20.983339    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 18:34:20.983480 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:20.983391    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Jul 28 18:34:20.990837 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:20.990817    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh"]
Jul 28 18:34:21.074045 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:21.073988    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5a88b7d-9567-4214-a36e-65f48804f27a-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh\" (UID: \"b5a88b7d-9567-4214-a36e-65f48804f27a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh"
Jul 28 18:34:21.074241 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:21.074121    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5a88b7d-9567-4214-a36e-65f48804f27a-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh\" (UID: \"b5a88b7d-9567-4214-a36e-65f48804f27a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh"
Jul 28 18:34:21.074241 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:21.074160    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5a88b7d-9567-4214-a36e-65f48804f27a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh\" (UID: \"b5a88b7d-9567-4214-a36e-65f48804f27a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh"
Jul 28 18:34:21.074241 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:21.074186    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7x6k\" (UniqueName: \"kubernetes.io/projected/b5a88b7d-9567-4214-a36e-65f48804f27a-kube-api-access-j7x6k\") pod \"isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh\" (UID: \"b5a88b7d-9567-4214-a36e-65f48804f27a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh"
Jul 28 18:34:21.174927 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:21.174875    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5a88b7d-9567-4214-a36e-65f48804f27a-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh\" (UID: \"b5a88b7d-9567-4214-a36e-65f48804f27a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh"
Jul 28 18:34:21.174927 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:21.174935    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5a88b7d-9567-4214-a36e-65f48804f27a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh\" (UID: \"b5a88b7d-9567-4214-a36e-65f48804f27a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh"
Jul 28 18:34:21.175170 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:21.174979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7x6k\" (UniqueName: \"kubernetes.io/projected/b5a88b7d-9567-4214-a36e-65f48804f27a-kube-api-access-j7x6k\") pod \"isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh\" (UID: \"b5a88b7d-9567-4214-a36e-65f48804f27a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh"
Jul 28 18:34:21.175170 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:21.175013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5a88b7d-9567-4214-a36e-65f48804f27a-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh\" (UID: \"b5a88b7d-9567-4214-a36e-65f48804f27a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh"
Jul 28 18:34:21.175170 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:34:21.175088    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-predictor-serving-cert: secret "isvc-sklearn-s3-predictor-serving-cert" not found
Jul 28 18:34:21.175170 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:34:21.175166    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5a88b7d-9567-4214-a36e-65f48804f27a-proxy-tls podName:b5a88b7d-9567-4214-a36e-65f48804f27a nodeName:}" failed. No retries permitted until 2026-07-28 18:34:21.675144896 +0000 UTC m=+5109.949761305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b5a88b7d-9567-4214-a36e-65f48804f27a-proxy-tls") pod "isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh" (UID: "b5a88b7d-9567-4214-a36e-65f48804f27a") : secret "isvc-sklearn-s3-predictor-serving-cert" not found
Jul 28 18:34:21.175407 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:21.175384    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5a88b7d-9567-4214-a36e-65f48804f27a-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh\" (UID: \"b5a88b7d-9567-4214-a36e-65f48804f27a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh"
Jul 28 18:34:21.175708 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:21.175685    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5a88b7d-9567-4214-a36e-65f48804f27a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh\" (UID: \"b5a88b7d-9567-4214-a36e-65f48804f27a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh"
Jul 28 18:34:21.183992 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:21.183970    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7x6k\" (UniqueName: \"kubernetes.io/projected/b5a88b7d-9567-4214-a36e-65f48804f27a-kube-api-access-j7x6k\") pod \"isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh\" (UID: \"b5a88b7d-9567-4214-a36e-65f48804f27a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh"
Jul 28 18:34:21.328531 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:21.328444    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:34:21.680276 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:21.680234    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5a88b7d-9567-4214-a36e-65f48804f27a-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh\" (UID: \"b5a88b7d-9567-4214-a36e-65f48804f27a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh"
Jul 28 18:34:21.682657 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:21.682634    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5a88b7d-9567-4214-a36e-65f48804f27a-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh\" (UID: \"b5a88b7d-9567-4214-a36e-65f48804f27a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh"
Jul 28 18:34:21.889961 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:21.889911    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh"
Jul 28 18:34:22.025304 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:22.025264    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh"]
Jul 28 18:34:22.028499 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:34:22.028465    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a88b7d_9567_4214_a36e_65f48804f27a.slice/crio-76d2bcc7b879f46fd1fd954ea1de28b7e75d2ac1d2e803d19511257eed7b3ddc WatchSource:0}: Error finding container 76d2bcc7b879f46fd1fd954ea1de28b7e75d2ac1d2e803d19511257eed7b3ddc: Status 404 returned error can't find the container with id 76d2bcc7b879f46fd1fd954ea1de28b7e75d2ac1d2e803d19511257eed7b3ddc
Jul 28 18:34:22.930989 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:22.930936    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh" event={"ID":"b5a88b7d-9567-4214-a36e-65f48804f27a","Type":"ContainerStarted","Data":"cba00047a6c721c337ec7a42800093701a6fcdd2726e9444d6ef23c91b7523eb"}
Jul 28 18:34:22.931189 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:22.931018    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh" event={"ID":"b5a88b7d-9567-4214-a36e-65f48804f27a","Type":"ContainerStarted","Data":"76d2bcc7b879f46fd1fd954ea1de28b7e75d2ac1d2e803d19511257eed7b3ddc"}
Jul 28 18:34:23.936281 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:23.936239    2581 generic.go:356] "Generic (PLEG): container finished" podID="b5a88b7d-9567-4214-a36e-65f48804f27a" containerID="cba00047a6c721c337ec7a42800093701a6fcdd2726e9444d6ef23c91b7523eb" exitCode=0
Jul 28 18:34:23.936779 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:23.936334    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh" event={"ID":"b5a88b7d-9567-4214-a36e-65f48804f27a","Type":"ContainerDied","Data":"cba00047a6c721c337ec7a42800093701a6fcdd2726e9444d6ef23c91b7523eb"}
Jul 28 18:34:24.942072 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:24.942033    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh" event={"ID":"b5a88b7d-9567-4214-a36e-65f48804f27a","Type":"ContainerStarted","Data":"b5143f6207082d9c282758978b1b05e04ea3dd066e713a6a37ba08771e289265"}
Jul 28 18:34:24.942466 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:24.942097    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh" event={"ID":"b5a88b7d-9567-4214-a36e-65f48804f27a","Type":"ContainerStarted","Data":"2d2f1c607b7207b5af9e0f02e1e2818e2d952acf837333581270011874295ed1"}
Jul 28 18:34:24.942466 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:24.942140    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh"
Jul 28 18:34:24.962255 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:24.962201    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh" podStartSLOduration=3.834644543 podStartE2EDuration="4.962179513s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.12753497s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:34:24.961108761 +0000 UTC m=+5113.235725193" watchObservedRunningTime="2026-07-28 18:34:24.962179513 +0000 UTC m=+5113.236795943"
Jul 28 18:34:25.945255 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:25.945219    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh"
Jul 28 18:34:25.946621 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:25.946587    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh" podUID="b5a88b7d-9567-4214-a36e-65f48804f27a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 28 18:34:26.948171 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:26.948124    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh" podUID="b5a88b7d-9567-4214-a36e-65f48804f27a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 28 18:34:31.953191 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:31.953156    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh"
Jul 28 18:34:31.953656 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:31.953628    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh" podUID="b5a88b7d-9567-4214-a36e-65f48804f27a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 28 18:34:41.954293 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:41.954197    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh" podUID="b5a88b7d-9567-4214-a36e-65f48804f27a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 28 18:34:51.954582 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:34:51.954535    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh" podUID="b5a88b7d-9567-4214-a36e-65f48804f27a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 28 18:35:01.954265 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:01.954224    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh" podUID="b5a88b7d-9567-4214-a36e-65f48804f27a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 28 18:35:11.954532 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:11.954491    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh" podUID="b5a88b7d-9567-4214-a36e-65f48804f27a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 28 18:35:21.953837 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:21.953797    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh" podUID="b5a88b7d-9567-4214-a36e-65f48804f27a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 28 18:35:31.954199 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:31.954145    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh" podUID="b5a88b7d-9567-4214-a36e-65f48804f27a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 28 18:35:41.954684 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:41.954646    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh"
Jul 28 18:35:51.113509 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.113476    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh"]
Jul 28 18:35:51.114026 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.113743    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh" podUID="b5a88b7d-9567-4214-a36e-65f48804f27a" containerName="kserve-container" containerID="cri-o://2d2f1c607b7207b5af9e0f02e1e2818e2d952acf837333581270011874295ed1" gracePeriod=30
Jul 28 18:35:51.114026 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.113758    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh" podUID="b5a88b7d-9567-4214-a36e-65f48804f27a" containerName="kube-rbac-proxy" containerID="cri-o://b5143f6207082d9c282758978b1b05e04ea3dd066e713a6a37ba08771e289265" gracePeriod=30
Jul 28 18:35:51.127877 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.127843    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:35:51.227687 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.227654    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:35:51.244327 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.244300    2581 generic.go:356] "Generic (PLEG): container finished" podID="b5a88b7d-9567-4214-a36e-65f48804f27a" containerID="b5143f6207082d9c282758978b1b05e04ea3dd066e713a6a37ba08771e289265" exitCode=2
Jul 28 18:35:51.244476 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.244379    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh" event={"ID":"b5a88b7d-9567-4214-a36e-65f48804f27a","Type":"ContainerDied","Data":"b5143f6207082d9c282758978b1b05e04ea3dd066e713a6a37ba08771e289265"}
Jul 28 18:35:51.246363 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.246336    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"]
Jul 28 18:35:51.249551 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.249535    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"
Jul 28 18:35:51.252205 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.252186    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:35:51.252301 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.252191    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Jul 28 18:35:51.252301 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.252192    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Jul 28 18:35:51.262929 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.262892    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"]
Jul 28 18:35:51.360605 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.360557    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq2t5\" (UniqueName: \"kubernetes.io/projected/83d8b07b-bd16-4785-9a04-e780e6ce7212-kube-api-access-tq2t5\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54\" (UID: \"83d8b07b-bd16-4785-9a04-e780e6ce7212\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"
Jul 28 18:35:51.360816 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.360628    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83d8b07b-bd16-4785-9a04-e780e6ce7212-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54\" (UID: \"83d8b07b-bd16-4785-9a04-e780e6ce7212\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"
Jul 28 18:35:51.360816 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.360661    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83d8b07b-bd16-4785-9a04-e780e6ce7212-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54\" (UID: \"83d8b07b-bd16-4785-9a04-e780e6ce7212\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"
Jul 28 18:35:51.360816 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.360704    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83d8b07b-bd16-4785-9a04-e780e6ce7212-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54\" (UID: \"83d8b07b-bd16-4785-9a04-e780e6ce7212\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"
Jul 28 18:35:51.360816 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.360789    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83d8b07b-bd16-4785-9a04-e780e6ce7212-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54\" (UID: \"83d8b07b-bd16-4785-9a04-e780e6ce7212\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"
Jul 28 18:35:51.462166 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.462123    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83d8b07b-bd16-4785-9a04-e780e6ce7212-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54\" (UID: \"83d8b07b-bd16-4785-9a04-e780e6ce7212\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"
Jul 28 18:35:51.462332 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.462179    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83d8b07b-bd16-4785-9a04-e780e6ce7212-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54\" (UID: \"83d8b07b-bd16-4785-9a04-e780e6ce7212\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"
Jul 28 18:35:51.462332 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.462225    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83d8b07b-bd16-4785-9a04-e780e6ce7212-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54\" (UID: \"83d8b07b-bd16-4785-9a04-e780e6ce7212\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"
Jul 28 18:35:51.462332 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.462267    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83d8b07b-bd16-4785-9a04-e780e6ce7212-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54\" (UID: \"83d8b07b-bd16-4785-9a04-e780e6ce7212\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"
Jul 28 18:35:51.462332 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.462309    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq2t5\" (UniqueName: \"kubernetes.io/projected/83d8b07b-bd16-4785-9a04-e780e6ce7212-kube-api-access-tq2t5\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54\" (UID: \"83d8b07b-bd16-4785-9a04-e780e6ce7212\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"
Jul 28 18:35:51.462540 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:35:51.462389    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Jul 28 18:35:51.462540 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:35:51.462473    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83d8b07b-bd16-4785-9a04-e780e6ce7212-proxy-tls podName:83d8b07b-bd16-4785-9a04-e780e6ce7212 nodeName:}" failed. No retries permitted until 2026-07-28 18:35:51.962452696 +0000 UTC m=+5200.237069109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/83d8b07b-bd16-4785-9a04-e780e6ce7212-proxy-tls") pod "isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54" (UID: "83d8b07b-bd16-4785-9a04-e780e6ce7212") : secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Jul 28 18:35:51.462762 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.462734    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83d8b07b-bd16-4785-9a04-e780e6ce7212-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54\" (UID: \"83d8b07b-bd16-4785-9a04-e780e6ce7212\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"
Jul 28 18:35:51.462917 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.462896    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83d8b07b-bd16-4785-9a04-e780e6ce7212-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54\" (UID: \"83d8b07b-bd16-4785-9a04-e780e6ce7212\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"
Jul 28 18:35:51.463112 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.463092    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83d8b07b-bd16-4785-9a04-e780e6ce7212-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54\" (UID: \"83d8b07b-bd16-4785-9a04-e780e6ce7212\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"
Jul 28 18:35:51.471385 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.471362    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq2t5\" (UniqueName: \"kubernetes.io/projected/83d8b07b-bd16-4785-9a04-e780e6ce7212-kube-api-access-tq2t5\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54\" (UID: \"83d8b07b-bd16-4785-9a04-e780e6ce7212\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"
Jul 28 18:35:51.686730 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.686691    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:35:51.948449 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.948397    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh" podUID="b5a88b7d-9567-4214-a36e-65f48804f27a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.69:8643/healthz\": dial tcp 10.133.0.69:8643: connect: connection refused"
Jul 28 18:35:51.953791 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.953752    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh" podUID="b5a88b7d-9567-4214-a36e-65f48804f27a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 28 18:35:51.966308 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.966275    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83d8b07b-bd16-4785-9a04-e780e6ce7212-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54\" (UID: \"83d8b07b-bd16-4785-9a04-e780e6ce7212\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"
Jul 28 18:35:51.968821 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:51.968795    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83d8b07b-bd16-4785-9a04-e780e6ce7212-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54\" (UID: \"83d8b07b-bd16-4785-9a04-e780e6ce7212\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"
Jul 28 18:35:52.160645 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:52.160605    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"
Jul 28 18:35:52.292972 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:52.292922    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"]
Jul 28 18:35:52.296118 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:35:52.296087    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d8b07b_bd16_4785_9a04_e780e6ce7212.slice/crio-972bd83e9fafb960bddeb29e7e20706df44449649a2689c4e7eb2430e13120a0 WatchSource:0}: Error finding container 972bd83e9fafb960bddeb29e7e20706df44449649a2689c4e7eb2430e13120a0: Status 404 returned error can't find the container with id 972bd83e9fafb960bddeb29e7e20706df44449649a2689c4e7eb2430e13120a0
Jul 28 18:35:53.254605 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:53.254564    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54" event={"ID":"83d8b07b-bd16-4785-9a04-e780e6ce7212","Type":"ContainerStarted","Data":"db9cc8184a0c92ba0e5dc0b4a01d9328ad12f14958dd8a11ee86c09039ef9d16"}
Jul 28 18:35:53.255151 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:53.254637    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54" event={"ID":"83d8b07b-bd16-4785-9a04-e780e6ce7212","Type":"ContainerStarted","Data":"972bd83e9fafb960bddeb29e7e20706df44449649a2689c4e7eb2430e13120a0"}
Jul 28 18:35:54.260123 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:54.260079    2581 generic.go:356] "Generic (PLEG): container finished" podID="83d8b07b-bd16-4785-9a04-e780e6ce7212" containerID="db9cc8184a0c92ba0e5dc0b4a01d9328ad12f14958dd8a11ee86c09039ef9d16" exitCode=0
Jul 28 18:35:54.260631 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:54.260172    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54" event={"ID":"83d8b07b-bd16-4785-9a04-e780e6ce7212","Type":"ContainerDied","Data":"db9cc8184a0c92ba0e5dc0b4a01d9328ad12f14958dd8a11ee86c09039ef9d16"}
Jul 28 18:35:55.266985 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:55.266926    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54" event={"ID":"83d8b07b-bd16-4785-9a04-e780e6ce7212","Type":"ContainerStarted","Data":"6604040ee6f9cbfc02e1ac26140f4e8e8c811558e24e559d681d8638d208619a"}
Jul 28 18:35:55.267379 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:55.267005    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54" event={"ID":"83d8b07b-bd16-4785-9a04-e780e6ce7212","Type":"ContainerStarted","Data":"9c7b45c2a4faca93558e526ce1e543c17a00ca39f9854c32f3120c46a8ed1e77"}
Jul 28 18:35:55.267379 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:55.267141    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"
Jul 28 18:35:56.268805 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.268776    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh"
Jul 28 18:35:56.272011 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.271979    2581 generic.go:356] "Generic (PLEG): container finished" podID="b5a88b7d-9567-4214-a36e-65f48804f27a" containerID="2d2f1c607b7207b5af9e0f02e1e2818e2d952acf837333581270011874295ed1" exitCode=0
Jul 28 18:35:56.272179 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.272025    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh"
Jul 28 18:35:56.272179 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.272020    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh" event={"ID":"b5a88b7d-9567-4214-a36e-65f48804f27a","Type":"ContainerDied","Data":"2d2f1c607b7207b5af9e0f02e1e2818e2d952acf837333581270011874295ed1"}
Jul 28 18:35:56.272302 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.272211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh" event={"ID":"b5a88b7d-9567-4214-a36e-65f48804f27a","Type":"ContainerDied","Data":"76d2bcc7b879f46fd1fd954ea1de28b7e75d2ac1d2e803d19511257eed7b3ddc"}
Jul 28 18:35:56.272302 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.272248    2581 scope.go:117] "RemoveContainer" containerID="b5143f6207082d9c282758978b1b05e04ea3dd066e713a6a37ba08771e289265"
Jul 28 18:35:56.272561 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.272542    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"
Jul 28 18:35:56.273919 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.273894    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54" podUID="83d8b07b-bd16-4785-9a04-e780e6ce7212" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Jul 28 18:35:56.281771 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.281162    2581 scope.go:117] "RemoveContainer" containerID="2d2f1c607b7207b5af9e0f02e1e2818e2d952acf837333581270011874295ed1"
Jul 28 18:35:56.290209 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.290181    2581 scope.go:117] "RemoveContainer" containerID="cba00047a6c721c337ec7a42800093701a6fcdd2726e9444d6ef23c91b7523eb"
Jul 28 18:35:56.290603 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.290556    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54" podStartSLOduration=4.206085596 podStartE2EDuration="5.290538389s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.084452793s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:35:55.288567808 +0000 UTC m=+5203.563184239" watchObservedRunningTime="2026-07-28 18:35:56.290538389 +0000 UTC m=+5204.565154823"
Jul 28 18:35:56.297536 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.297514    2581 scope.go:117] "RemoveContainer" containerID="b5143f6207082d9c282758978b1b05e04ea3dd066e713a6a37ba08771e289265"
Jul 28 18:35:56.297825 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:35:56.297805    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5143f6207082d9c282758978b1b05e04ea3dd066e713a6a37ba08771e289265\": container with ID starting with b5143f6207082d9c282758978b1b05e04ea3dd066e713a6a37ba08771e289265 not found: ID does not exist" containerID="b5143f6207082d9c282758978b1b05e04ea3dd066e713a6a37ba08771e289265"
Jul 28 18:35:56.297892 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.297834    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5143f6207082d9c282758978b1b05e04ea3dd066e713a6a37ba08771e289265"} err="failed to get container status \"b5143f6207082d9c282758978b1b05e04ea3dd066e713a6a37ba08771e289265\": rpc error: code = NotFound desc = could not find container \"b5143f6207082d9c282758978b1b05e04ea3dd066e713a6a37ba08771e289265\": container with ID starting with b5143f6207082d9c282758978b1b05e04ea3dd066e713a6a37ba08771e289265 not found: ID does not exist"
Jul 28 18:35:56.297892 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.297852    2581 scope.go:117] "RemoveContainer" containerID="2d2f1c607b7207b5af9e0f02e1e2818e2d952acf837333581270011874295ed1"
Jul 28 18:35:56.298148 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:35:56.298130    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d2f1c607b7207b5af9e0f02e1e2818e2d952acf837333581270011874295ed1\": container with ID starting with 2d2f1c607b7207b5af9e0f02e1e2818e2d952acf837333581270011874295ed1 not found: ID does not exist" containerID="2d2f1c607b7207b5af9e0f02e1e2818e2d952acf837333581270011874295ed1"
Jul 28 18:35:56.298215 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.298155    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d2f1c607b7207b5af9e0f02e1e2818e2d952acf837333581270011874295ed1"} err="failed to get container status \"2d2f1c607b7207b5af9e0f02e1e2818e2d952acf837333581270011874295ed1\": rpc error: code = NotFound desc = could not find container \"2d2f1c607b7207b5af9e0f02e1e2818e2d952acf837333581270011874295ed1\": container with ID starting with 2d2f1c607b7207b5af9e0f02e1e2818e2d952acf837333581270011874295ed1 not found: ID does not exist"
Jul 28 18:35:56.298215 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.298172    2581 scope.go:117] "RemoveContainer" containerID="cba00047a6c721c337ec7a42800093701a6fcdd2726e9444d6ef23c91b7523eb"
Jul 28 18:35:56.298397 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:35:56.298382    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cba00047a6c721c337ec7a42800093701a6fcdd2726e9444d6ef23c91b7523eb\": container with ID starting with cba00047a6c721c337ec7a42800093701a6fcdd2726e9444d6ef23c91b7523eb not found: ID does not exist" containerID="cba00047a6c721c337ec7a42800093701a6fcdd2726e9444d6ef23c91b7523eb"
Jul 28 18:35:56.298442 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.298400    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cba00047a6c721c337ec7a42800093701a6fcdd2726e9444d6ef23c91b7523eb"} err="failed to get container status \"cba00047a6c721c337ec7a42800093701a6fcdd2726e9444d6ef23c91b7523eb\": rpc error: code = NotFound desc = could not find container \"cba00047a6c721c337ec7a42800093701a6fcdd2726e9444d6ef23c91b7523eb\": container with ID starting with cba00047a6c721c337ec7a42800093701a6fcdd2726e9444d6ef23c91b7523eb not found: ID does not exist"
Jul 28 18:35:56.300637 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.300621    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5a88b7d-9567-4214-a36e-65f48804f27a-kube-rbac-proxy-sar-config\") pod \"b5a88b7d-9567-4214-a36e-65f48804f27a\" (UID: \"b5a88b7d-9567-4214-a36e-65f48804f27a\") "
Jul 28 18:35:56.300719 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.300676    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5a88b7d-9567-4214-a36e-65f48804f27a-proxy-tls\") pod \"b5a88b7d-9567-4214-a36e-65f48804f27a\" (UID: \"b5a88b7d-9567-4214-a36e-65f48804f27a\") "
Jul 28 18:35:56.300719 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.300711    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7x6k\" (UniqueName: \"kubernetes.io/projected/b5a88b7d-9567-4214-a36e-65f48804f27a-kube-api-access-j7x6k\") pod \"b5a88b7d-9567-4214-a36e-65f48804f27a\" (UID: \"b5a88b7d-9567-4214-a36e-65f48804f27a\") "
Jul 28 18:35:56.300831 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.300731    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5a88b7d-9567-4214-a36e-65f48804f27a-kserve-provision-location\") pod \"b5a88b7d-9567-4214-a36e-65f48804f27a\" (UID: \"b5a88b7d-9567-4214-a36e-65f48804f27a\") "
Jul 28 18:35:56.301125 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.300991    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a88b7d-9567-4214-a36e-65f48804f27a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b5a88b7d-9567-4214-a36e-65f48804f27a" (UID: "b5a88b7d-9567-4214-a36e-65f48804f27a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:35:56.301125 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.301104    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a88b7d-9567-4214-a36e-65f48804f27a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b5a88b7d-9567-4214-a36e-65f48804f27a" (UID: "b5a88b7d-9567-4214-a36e-65f48804f27a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:35:56.302858 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.302830    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a88b7d-9567-4214-a36e-65f48804f27a-kube-api-access-j7x6k" (OuterVolumeSpecName: "kube-api-access-j7x6k") pod "b5a88b7d-9567-4214-a36e-65f48804f27a" (UID: "b5a88b7d-9567-4214-a36e-65f48804f27a"). InnerVolumeSpecName "kube-api-access-j7x6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:35:56.302969 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.302893    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a88b7d-9567-4214-a36e-65f48804f27a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b5a88b7d-9567-4214-a36e-65f48804f27a" (UID: "b5a88b7d-9567-4214-a36e-65f48804f27a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:35:56.401412 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.401372    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5a88b7d-9567-4214-a36e-65f48804f27a-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:35:56.401412 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.401410    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7x6k\" (UniqueName: \"kubernetes.io/projected/b5a88b7d-9567-4214-a36e-65f48804f27a-kube-api-access-j7x6k\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:35:56.401631 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.401425    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5a88b7d-9567-4214-a36e-65f48804f27a-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:35:56.401631 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.401437    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5a88b7d-9567-4214-a36e-65f48804f27a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:35:56.594658 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.594615    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh"]
Jul 28 18:35:56.600585 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:56.600559    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7d5449bb4f-qhjfh"]
Jul 28 18:35:57.276921 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:57.276887    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54" podUID="83d8b07b-bd16-4785-9a04-e780e6ce7212" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Jul 28 18:35:58.289297 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:35:58.289269    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5a88b7d-9567-4214-a36e-65f48804f27a" path="/var/lib/kubelet/pods/b5a88b7d-9567-4214-a36e-65f48804f27a/volumes"
Jul 28 18:36:02.280775 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:36:02.280745    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"
Jul 28 18:36:02.281305 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:36:02.281277    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54" podUID="83d8b07b-bd16-4785-9a04-e780e6ce7212" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Jul 28 18:36:12.282001 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:36:12.281882    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54" podUID="83d8b07b-bd16-4785-9a04-e780e6ce7212" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Jul 28 18:36:17.827107 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:36:17.827060    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:36:22.282146 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:36:22.282106    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54" podUID="83d8b07b-bd16-4785-9a04-e780e6ce7212" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Jul 28 18:36:32.281796 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:36:32.281747    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54" podUID="83d8b07b-bd16-4785-9a04-e780e6ce7212" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Jul 28 18:36:42.282159 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:36:42.282118    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54" podUID="83d8b07b-bd16-4785-9a04-e780e6ce7212" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Jul 28 18:36:52.281291 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:36:52.281186    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54" podUID="83d8b07b-bd16-4785-9a04-e780e6ce7212" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Jul 28 18:37:02.281273 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:02.281230    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54" podUID="83d8b07b-bd16-4785-9a04-e780e6ce7212" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Jul 28 18:37:12.282130 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:12.282090    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"
Jul 28 18:37:21.363513 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:21.363470    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"]
Jul 28 18:37:21.363996 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:21.363768    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54" podUID="83d8b07b-bd16-4785-9a04-e780e6ce7212" containerName="kserve-container" containerID="cri-o://9c7b45c2a4faca93558e526ce1e543c17a00ca39f9854c32f3120c46a8ed1e77" gracePeriod=30
Jul 28 18:37:21.363996 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:21.363834    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54" podUID="83d8b07b-bd16-4785-9a04-e780e6ce7212" containerName="kube-rbac-proxy" containerID="cri-o://6604040ee6f9cbfc02e1ac26140f4e8e8c811558e24e559d681d8638d208619a" gracePeriod=30
Jul 28 18:37:21.383704 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:21.383675    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:37:21.570250 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:21.570215    2581 generic.go:356] "Generic (PLEG): container finished" podID="83d8b07b-bd16-4785-9a04-e780e6ce7212" containerID="6604040ee6f9cbfc02e1ac26140f4e8e8c811558e24e559d681d8638d208619a" exitCode=2
Jul 28 18:37:21.570413 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:21.570285    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54" event={"ID":"83d8b07b-bd16-4785-9a04-e780e6ce7212","Type":"ContainerDied","Data":"6604040ee6f9cbfc02e1ac26140f4e8e8c811558e24e559d681d8638d208619a"}
Jul 28 18:37:22.278046 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.277993    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54" podUID="83d8b07b-bd16-4785-9a04-e780e6ce7212" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.70:8643/healthz\": dial tcp 10.133.0.70:8643: connect: connection refused"
Jul 28 18:37:22.281246 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.281216    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54" podUID="83d8b07b-bd16-4785-9a04-e780e6ce7212" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Jul 28 18:37:22.406438 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.406403    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:37:22.444293 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.444255    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9"]
Jul 28 18:37:22.444532 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.444519    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5a88b7d-9567-4214-a36e-65f48804f27a" containerName="kube-rbac-proxy"
Jul 28 18:37:22.444574 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.444533    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5a88b7d-9567-4214-a36e-65f48804f27a" containerName="kube-rbac-proxy"
Jul 28 18:37:22.444574 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.444546    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5a88b7d-9567-4214-a36e-65f48804f27a" containerName="storage-initializer"
Jul 28 18:37:22.444574 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.444553    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5a88b7d-9567-4214-a36e-65f48804f27a" containerName="storage-initializer"
Jul 28 18:37:22.444574 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.444564    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5a88b7d-9567-4214-a36e-65f48804f27a" containerName="kserve-container"
Jul 28 18:37:22.444574 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.444571    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5a88b7d-9567-4214-a36e-65f48804f27a" containerName="kserve-container"
Jul 28 18:37:22.444735 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.444618    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5a88b7d-9567-4214-a36e-65f48804f27a" containerName="kube-rbac-proxy"
Jul 28 18:37:22.444735 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.444627    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5a88b7d-9567-4214-a36e-65f48804f27a" containerName="kserve-container"
Jul 28 18:37:22.447618 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.447598    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9"
Jul 28 18:37:22.450631 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.450607    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Jul 28 18:37:22.450777 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.450607    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Jul 28 18:37:22.461246 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.461215    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9"]
Jul 28 18:37:22.549697 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.549598    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:37:22.553669 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.553648    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp8lm\" (UniqueName: \"kubernetes.io/projected/cc5dd1f2-6cfa-45b1-886a-883f1a72c75a-kube-api-access-bp8lm\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9\" (UID: \"cc5dd1f2-6cfa-45b1-886a-883f1a72c75a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9"
Jul 28 18:37:22.553751 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.553681    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc5dd1f2-6cfa-45b1-886a-883f1a72c75a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9\" (UID: \"cc5dd1f2-6cfa-45b1-886a-883f1a72c75a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9"
Jul 28 18:37:22.553751 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.553714    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc5dd1f2-6cfa-45b1-886a-883f1a72c75a-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9\" (UID: \"cc5dd1f2-6cfa-45b1-886a-883f1a72c75a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9"
Jul 28 18:37:22.553838 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.553800    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc5dd1f2-6cfa-45b1-886a-883f1a72c75a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9\" (UID: \"cc5dd1f2-6cfa-45b1-886a-883f1a72c75a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9"
Jul 28 18:37:22.654847 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.654800    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc5dd1f2-6cfa-45b1-886a-883f1a72c75a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9\" (UID: \"cc5dd1f2-6cfa-45b1-886a-883f1a72c75a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9"
Jul 28 18:37:22.654847 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.654854    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp8lm\" (UniqueName: \"kubernetes.io/projected/cc5dd1f2-6cfa-45b1-886a-883f1a72c75a-kube-api-access-bp8lm\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9\" (UID: \"cc5dd1f2-6cfa-45b1-886a-883f1a72c75a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9"
Jul 28 18:37:22.655175 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.654882    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc5dd1f2-6cfa-45b1-886a-883f1a72c75a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9\" (UID: \"cc5dd1f2-6cfa-45b1-886a-883f1a72c75a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9"
Jul 28 18:37:22.655175 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.654914    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc5dd1f2-6cfa-45b1-886a-883f1a72c75a-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9\" (UID: \"cc5dd1f2-6cfa-45b1-886a-883f1a72c75a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9"
Jul 28 18:37:22.655349 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.655326    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc5dd1f2-6cfa-45b1-886a-883f1a72c75a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9\" (UID: \"cc5dd1f2-6cfa-45b1-886a-883f1a72c75a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9"
Jul 28 18:37:22.655606 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.655580    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc5dd1f2-6cfa-45b1-886a-883f1a72c75a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9\" (UID: \"cc5dd1f2-6cfa-45b1-886a-883f1a72c75a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9"
Jul 28 18:37:22.657593 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.657563    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc5dd1f2-6cfa-45b1-886a-883f1a72c75a-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9\" (UID: \"cc5dd1f2-6cfa-45b1-886a-883f1a72c75a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9"
Jul 28 18:37:22.664322 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.664293    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp8lm\" (UniqueName: \"kubernetes.io/projected/cc5dd1f2-6cfa-45b1-886a-883f1a72c75a-kube-api-access-bp8lm\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9\" (UID: \"cc5dd1f2-6cfa-45b1-886a-883f1a72c75a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9"
Jul 28 18:37:22.757516 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.757467    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9"
Jul 28 18:37:22.887758 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:22.887733    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9"]
Jul 28 18:37:22.890277 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:37:22.890252    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5dd1f2_6cfa_45b1_886a_883f1a72c75a.slice/crio-fcd2f874a67092f0b9a6706aeaf0ce2f910904070e5d86ee6a13223ca1ac2b00 WatchSource:0}: Error finding container fcd2f874a67092f0b9a6706aeaf0ce2f910904070e5d86ee6a13223ca1ac2b00: Status 404 returned error can't find the container with id fcd2f874a67092f0b9a6706aeaf0ce2f910904070e5d86ee6a13223ca1ac2b00
Jul 28 18:37:23.579577 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:23.579535    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9" event={"ID":"cc5dd1f2-6cfa-45b1-886a-883f1a72c75a","Type":"ContainerStarted","Data":"e02a7e0e8dd5eabc1eb889e8e4b6566ff3a33b7fb254fc0fbea4440e452eec9c"}
Jul 28 18:37:23.580093 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:23.579640    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9" event={"ID":"cc5dd1f2-6cfa-45b1-886a-883f1a72c75a","Type":"ContainerStarted","Data":"fcd2f874a67092f0b9a6706aeaf0ce2f910904070e5d86ee6a13223ca1ac2b00"}
Jul 28 18:37:25.587913 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:25.587885    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9_cc5dd1f2-6cfa-45b1-886a-883f1a72c75a/storage-initializer/0.log"
Jul 28 18:37:25.588368 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:25.587928    2581 generic.go:356] "Generic (PLEG): container finished" podID="cc5dd1f2-6cfa-45b1-886a-883f1a72c75a" containerID="e02a7e0e8dd5eabc1eb889e8e4b6566ff3a33b7fb254fc0fbea4440e452eec9c" exitCode=1
Jul 28 18:37:25.588368 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:25.587986    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9" event={"ID":"cc5dd1f2-6cfa-45b1-886a-883f1a72c75a","Type":"ContainerDied","Data":"e02a7e0e8dd5eabc1eb889e8e4b6566ff3a33b7fb254fc0fbea4440e452eec9c"}
Jul 28 18:37:26.307261 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.307221    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"
Jul 28 18:37:26.385095 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.385055    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83d8b07b-bd16-4785-9a04-e780e6ce7212-cabundle-cert\") pod \"83d8b07b-bd16-4785-9a04-e780e6ce7212\" (UID: \"83d8b07b-bd16-4785-9a04-e780e6ce7212\") "
Jul 28 18:37:26.385275 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.385115    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83d8b07b-bd16-4785-9a04-e780e6ce7212-proxy-tls\") pod \"83d8b07b-bd16-4785-9a04-e780e6ce7212\" (UID: \"83d8b07b-bd16-4785-9a04-e780e6ce7212\") "
Jul 28 18:37:26.385275 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.385163    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83d8b07b-bd16-4785-9a04-e780e6ce7212-kube-rbac-proxy-sar-config\") pod \"83d8b07b-bd16-4785-9a04-e780e6ce7212\" (UID: \"83d8b07b-bd16-4785-9a04-e780e6ce7212\") "
Jul 28 18:37:26.385275 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.385196    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq2t5\" (UniqueName: \"kubernetes.io/projected/83d8b07b-bd16-4785-9a04-e780e6ce7212-kube-api-access-tq2t5\") pod \"83d8b07b-bd16-4785-9a04-e780e6ce7212\" (UID: \"83d8b07b-bd16-4785-9a04-e780e6ce7212\") "
Jul 28 18:37:26.385275 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.385223    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83d8b07b-bd16-4785-9a04-e780e6ce7212-kserve-provision-location\") pod \"83d8b07b-bd16-4785-9a04-e780e6ce7212\" (UID: \"83d8b07b-bd16-4785-9a04-e780e6ce7212\") "
Jul 28 18:37:26.385501 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.385471    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83d8b07b-bd16-4785-9a04-e780e6ce7212-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "83d8b07b-bd16-4785-9a04-e780e6ce7212" (UID: "83d8b07b-bd16-4785-9a04-e780e6ce7212"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:37:26.385707 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.385682    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d8b07b-bd16-4785-9a04-e780e6ce7212-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "83d8b07b-bd16-4785-9a04-e780e6ce7212" (UID: "83d8b07b-bd16-4785-9a04-e780e6ce7212"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:37:26.385797 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.385751    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83d8b07b-bd16-4785-9a04-e780e6ce7212-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "83d8b07b-bd16-4785-9a04-e780e6ce7212" (UID: "83d8b07b-bd16-4785-9a04-e780e6ce7212"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:37:26.388146 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.388118    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83d8b07b-bd16-4785-9a04-e780e6ce7212-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "83d8b07b-bd16-4785-9a04-e780e6ce7212" (UID: "83d8b07b-bd16-4785-9a04-e780e6ce7212"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:37:26.388252 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.388129    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83d8b07b-bd16-4785-9a04-e780e6ce7212-kube-api-access-tq2t5" (OuterVolumeSpecName: "kube-api-access-tq2t5") pod "83d8b07b-bd16-4785-9a04-e780e6ce7212" (UID: "83d8b07b-bd16-4785-9a04-e780e6ce7212"). InnerVolumeSpecName "kube-api-access-tq2t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:37:26.486537 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.486489    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83d8b07b-bd16-4785-9a04-e780e6ce7212-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:37:26.486537 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.486525    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83d8b07b-bd16-4785-9a04-e780e6ce7212-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:37:26.486537 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.486537    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq2t5\" (UniqueName: \"kubernetes.io/projected/83d8b07b-bd16-4785-9a04-e780e6ce7212-kube-api-access-tq2t5\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:37:26.486537 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.486545    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83d8b07b-bd16-4785-9a04-e780e6ce7212-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:37:26.486819 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.486555    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83d8b07b-bd16-4785-9a04-e780e6ce7212-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:37:26.593227 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.593196    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9_cc5dd1f2-6cfa-45b1-886a-883f1a72c75a/storage-initializer/0.log"
Jul 28 18:37:26.593682 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.593297    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9" event={"ID":"cc5dd1f2-6cfa-45b1-886a-883f1a72c75a","Type":"ContainerStarted","Data":"300e711781f4e083270192acbf30f19dab869c38ccfe6e285c9c3e32c1eb1aea"}
Jul 28 18:37:26.595111 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.595082    2581 generic.go:356] "Generic (PLEG): container finished" podID="83d8b07b-bd16-4785-9a04-e780e6ce7212" containerID="9c7b45c2a4faca93558e526ce1e543c17a00ca39f9854c32f3120c46a8ed1e77" exitCode=0
Jul 28 18:37:26.595229 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.595128    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"
Jul 28 18:37:26.595229 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.595162    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54" event={"ID":"83d8b07b-bd16-4785-9a04-e780e6ce7212","Type":"ContainerDied","Data":"9c7b45c2a4faca93558e526ce1e543c17a00ca39f9854c32f3120c46a8ed1e77"}
Jul 28 18:37:26.595229 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.595218    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54" event={"ID":"83d8b07b-bd16-4785-9a04-e780e6ce7212","Type":"ContainerDied","Data":"972bd83e9fafb960bddeb29e7e20706df44449649a2689c4e7eb2430e13120a0"}
Jul 28 18:37:26.595331 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.595245    2581 scope.go:117] "RemoveContainer" containerID="6604040ee6f9cbfc02e1ac26140f4e8e8c811558e24e559d681d8638d208619a"
Jul 28 18:37:26.603274 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.603157    2581 scope.go:117] "RemoveContainer" containerID="9c7b45c2a4faca93558e526ce1e543c17a00ca39f9854c32f3120c46a8ed1e77"
Jul 28 18:37:26.610148 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.610131    2581 scope.go:117] "RemoveContainer" containerID="db9cc8184a0c92ba0e5dc0b4a01d9328ad12f14958dd8a11ee86c09039ef9d16"
Jul 28 18:37:26.617667 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.617649    2581 scope.go:117] "RemoveContainer" containerID="6604040ee6f9cbfc02e1ac26140f4e8e8c811558e24e559d681d8638d208619a"
Jul 28 18:37:26.617935 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:37:26.617912    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6604040ee6f9cbfc02e1ac26140f4e8e8c811558e24e559d681d8638d208619a\": container with ID starting with 6604040ee6f9cbfc02e1ac26140f4e8e8c811558e24e559d681d8638d208619a not found: ID does not exist" containerID="6604040ee6f9cbfc02e1ac26140f4e8e8c811558e24e559d681d8638d208619a"
Jul 28 18:37:26.618012 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.617968    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6604040ee6f9cbfc02e1ac26140f4e8e8c811558e24e559d681d8638d208619a"} err="failed to get container status \"6604040ee6f9cbfc02e1ac26140f4e8e8c811558e24e559d681d8638d208619a\": rpc error: code = NotFound desc = could not find container \"6604040ee6f9cbfc02e1ac26140f4e8e8c811558e24e559d681d8638d208619a\": container with ID starting with 6604040ee6f9cbfc02e1ac26140f4e8e8c811558e24e559d681d8638d208619a not found: ID does not exist"
Jul 28 18:37:26.618012 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.617986    2581 scope.go:117] "RemoveContainer" containerID="9c7b45c2a4faca93558e526ce1e543c17a00ca39f9854c32f3120c46a8ed1e77"
Jul 28 18:37:26.618250 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:37:26.618231    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c7b45c2a4faca93558e526ce1e543c17a00ca39f9854c32f3120c46a8ed1e77\": container with ID starting with 9c7b45c2a4faca93558e526ce1e543c17a00ca39f9854c32f3120c46a8ed1e77 not found: ID does not exist" containerID="9c7b45c2a4faca93558e526ce1e543c17a00ca39f9854c32f3120c46a8ed1e77"
Jul 28 18:37:26.618295 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.618258    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c7b45c2a4faca93558e526ce1e543c17a00ca39f9854c32f3120c46a8ed1e77"} err="failed to get container status \"9c7b45c2a4faca93558e526ce1e543c17a00ca39f9854c32f3120c46a8ed1e77\": rpc error: code = NotFound desc = could not find container \"9c7b45c2a4faca93558e526ce1e543c17a00ca39f9854c32f3120c46a8ed1e77\": container with ID starting with 9c7b45c2a4faca93558e526ce1e543c17a00ca39f9854c32f3120c46a8ed1e77 not found: ID does not exist"
Jul 28 18:37:26.618295 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.618274    2581 scope.go:117] "RemoveContainer" containerID="db9cc8184a0c92ba0e5dc0b4a01d9328ad12f14958dd8a11ee86c09039ef9d16"
Jul 28 18:37:26.618492 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:37:26.618473    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db9cc8184a0c92ba0e5dc0b4a01d9328ad12f14958dd8a11ee86c09039ef9d16\": container with ID starting with db9cc8184a0c92ba0e5dc0b4a01d9328ad12f14958dd8a11ee86c09039ef9d16 not found: ID does not exist" containerID="db9cc8184a0c92ba0e5dc0b4a01d9328ad12f14958dd8a11ee86c09039ef9d16"
Jul 28 18:37:26.618532 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.618497    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db9cc8184a0c92ba0e5dc0b4a01d9328ad12f14958dd8a11ee86c09039ef9d16"} err="failed to get container status \"db9cc8184a0c92ba0e5dc0b4a01d9328ad12f14958dd8a11ee86c09039ef9d16\": rpc error: code = NotFound desc = could not find container \"db9cc8184a0c92ba0e5dc0b4a01d9328ad12f14958dd8a11ee86c09039ef9d16\": container with ID starting with db9cc8184a0c92ba0e5dc0b4a01d9328ad12f14958dd8a11ee86c09039ef9d16 not found: ID does not exist"
Jul 28 18:37:26.625189 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.625164    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"]
Jul 28 18:37:26.628852 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:26.628822    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-8hx54"]
Jul 28 18:37:28.289579 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:28.289548    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83d8b07b-bd16-4785-9a04-e780e6ce7212" path="/var/lib/kubelet/pods/83d8b07b-bd16-4785-9a04-e780e6ce7212/volumes"
Jul 28 18:37:29.613880 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:29.613851    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9_cc5dd1f2-6cfa-45b1-886a-883f1a72c75a/storage-initializer/1.log"
Jul 28 18:37:29.614327 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:29.614231    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9_cc5dd1f2-6cfa-45b1-886a-883f1a72c75a/storage-initializer/0.log"
Jul 28 18:37:29.614327 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:29.614259    2581 generic.go:356] "Generic (PLEG): container finished" podID="cc5dd1f2-6cfa-45b1-886a-883f1a72c75a" containerID="300e711781f4e083270192acbf30f19dab869c38ccfe6e285c9c3e32c1eb1aea" exitCode=1
Jul 28 18:37:29.614423 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:29.614339    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9" event={"ID":"cc5dd1f2-6cfa-45b1-886a-883f1a72c75a","Type":"ContainerDied","Data":"300e711781f4e083270192acbf30f19dab869c38ccfe6e285c9c3e32c1eb1aea"}
Jul 28 18:37:29.614460 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:29.614436    2581 scope.go:117] "RemoveContainer" containerID="e02a7e0e8dd5eabc1eb889e8e4b6566ff3a33b7fb254fc0fbea4440e452eec9c"
Jul 28 18:37:29.614723 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:29.614697    2581 scope.go:117] "RemoveContainer" containerID="e02a7e0e8dd5eabc1eb889e8e4b6566ff3a33b7fb254fc0fbea4440e452eec9c"
Jul 28 18:37:29.624687 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:37:29.624664    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9_kserve-ci-e2e-test(cc5dd1f2-6cfa-45b1-886a-883f1a72c75a)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9" podUID="cc5dd1f2-6cfa-45b1-886a-883f1a72c75a"
Jul 28 18:37:29.624840 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:37:29.624813    2581 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9_kserve-ci-e2e-test_cc5dd1f2-6cfa-45b1-886a-883f1a72c75a_0 in pod sandbox fcd2f874a67092f0b9a6706aeaf0ce2f910904070e5d86ee6a13223ca1ac2b00 from index: no such id: 'e02a7e0e8dd5eabc1eb889e8e4b6566ff3a33b7fb254fc0fbea4440e452eec9c'" containerID="e02a7e0e8dd5eabc1eb889e8e4b6566ff3a33b7fb254fc0fbea4440e452eec9c"
Jul 28 18:37:29.624883 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:29.624848    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e02a7e0e8dd5eabc1eb889e8e4b6566ff3a33b7fb254fc0fbea4440e452eec9c"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9_kserve-ci-e2e-test_cc5dd1f2-6cfa-45b1-886a-883f1a72c75a_0 in pod sandbox fcd2f874a67092f0b9a6706aeaf0ce2f910904070e5d86ee6a13223ca1ac2b00 from index: no such id: 'e02a7e0e8dd5eabc1eb889e8e4b6566ff3a33b7fb254fc0fbea4440e452eec9c'"
Jul 28 18:37:30.619540 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:30.619502    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9_cc5dd1f2-6cfa-45b1-886a-883f1a72c75a/storage-initializer/1.log"
Jul 28 18:37:32.422114 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:32.422056    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9"]
Jul 28 18:37:32.444144 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:32.444108    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:37:32.550473 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:32.550440    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9_cc5dd1f2-6cfa-45b1-886a-883f1a72c75a/storage-initializer/1.log"
Jul 28 18:37:32.554077 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:32.554052    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9"
Jul 28 18:37:32.627552 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:32.627526    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9_cc5dd1f2-6cfa-45b1-886a-883f1a72c75a/storage-initializer/1.log"
Jul 28 18:37:32.627724 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:32.627586    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9" event={"ID":"cc5dd1f2-6cfa-45b1-886a-883f1a72c75a","Type":"ContainerDied","Data":"fcd2f874a67092f0b9a6706aeaf0ce2f910904070e5d86ee6a13223ca1ac2b00"}
Jul 28 18:37:32.627724 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:32.627617    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9"
Jul 28 18:37:32.627724 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:32.627638    2581 scope.go:117] "RemoveContainer" containerID="300e711781f4e083270192acbf30f19dab869c38ccfe6e285c9c3e32c1eb1aea"
Jul 28 18:37:32.635016 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:32.634985    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc5dd1f2-6cfa-45b1-886a-883f1a72c75a-proxy-tls\") pod \"cc5dd1f2-6cfa-45b1-886a-883f1a72c75a\" (UID: \"cc5dd1f2-6cfa-45b1-886a-883f1a72c75a\") "
Jul 28 18:37:32.635183 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:32.635048    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bp8lm\" (UniqueName: \"kubernetes.io/projected/cc5dd1f2-6cfa-45b1-886a-883f1a72c75a-kube-api-access-bp8lm\") pod \"cc5dd1f2-6cfa-45b1-886a-883f1a72c75a\" (UID: \"cc5dd1f2-6cfa-45b1-886a-883f1a72c75a\") "
Jul 28 18:37:32.635183 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:32.635139    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc5dd1f2-6cfa-45b1-886a-883f1a72c75a-kserve-provision-location\") pod \"cc5dd1f2-6cfa-45b1-886a-883f1a72c75a\" (UID: \"cc5dd1f2-6cfa-45b1-886a-883f1a72c75a\") "
Jul 28 18:37:32.635310 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:32.635208    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc5dd1f2-6cfa-45b1-886a-883f1a72c75a-kube-rbac-proxy-sar-config\") pod \"cc5dd1f2-6cfa-45b1-886a-883f1a72c75a\" (UID: \"cc5dd1f2-6cfa-45b1-886a-883f1a72c75a\") "
Jul 28 18:37:32.635438 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:32.635405    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5dd1f2-6cfa-45b1-886a-883f1a72c75a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cc5dd1f2-6cfa-45b1-886a-883f1a72c75a" (UID: "cc5dd1f2-6cfa-45b1-886a-883f1a72c75a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:37:32.635599 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:32.635576    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc5dd1f2-6cfa-45b1-886a-883f1a72c75a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cc5dd1f2-6cfa-45b1-886a-883f1a72c75a" (UID: "cc5dd1f2-6cfa-45b1-886a-883f1a72c75a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:37:32.637088 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:32.637060    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc5dd1f2-6cfa-45b1-886a-883f1a72c75a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cc5dd1f2-6cfa-45b1-886a-883f1a72c75a" (UID: "cc5dd1f2-6cfa-45b1-886a-883f1a72c75a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:37:32.637173 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:32.637124    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc5dd1f2-6cfa-45b1-886a-883f1a72c75a-kube-api-access-bp8lm" (OuterVolumeSpecName: "kube-api-access-bp8lm") pod "cc5dd1f2-6cfa-45b1-886a-883f1a72c75a" (UID: "cc5dd1f2-6cfa-45b1-886a-883f1a72c75a"). InnerVolumeSpecName "kube-api-access-bp8lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:37:32.736819 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:32.736716    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp8lm\" (UniqueName: \"kubernetes.io/projected/cc5dd1f2-6cfa-45b1-886a-883f1a72c75a-kube-api-access-bp8lm\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:37:32.736819 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:32.736762    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc5dd1f2-6cfa-45b1-886a-883f1a72c75a-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:37:32.736819 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:32.736772    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc5dd1f2-6cfa-45b1-886a-883f1a72c75a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:37:32.736819 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:32.736783    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc5dd1f2-6cfa-45b1-886a-883f1a72c75a-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:37:32.965362 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:32.965331    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9"]
Jul 28 18:37:32.969377 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:32.969350    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-8645544777-8wcq9"]
Jul 28 18:37:33.498718 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.498682    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:37:33.531770 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.531731    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"]
Jul 28 18:37:33.532073 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.532056    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc5dd1f2-6cfa-45b1-886a-883f1a72c75a" containerName="storage-initializer"
Jul 28 18:37:33.532166 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.532076    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc5dd1f2-6cfa-45b1-886a-883f1a72c75a" containerName="storage-initializer"
Jul 28 18:37:33.532166 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.532092    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83d8b07b-bd16-4785-9a04-e780e6ce7212" containerName="kserve-container"
Jul 28 18:37:33.532166 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.532102    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="83d8b07b-bd16-4785-9a04-e780e6ce7212" containerName="kserve-container"
Jul 28 18:37:33.532166 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.532113    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83d8b07b-bd16-4785-9a04-e780e6ce7212" containerName="storage-initializer"
Jul 28 18:37:33.532166 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.532122    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="83d8b07b-bd16-4785-9a04-e780e6ce7212" containerName="storage-initializer"
Jul 28 18:37:33.532166 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.532144    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83d8b07b-bd16-4785-9a04-e780e6ce7212" containerName="kube-rbac-proxy"
Jul 28 18:37:33.532166 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.532152    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="83d8b07b-bd16-4785-9a04-e780e6ce7212" containerName="kube-rbac-proxy"
Jul 28 18:37:33.532532 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.532214    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="83d8b07b-bd16-4785-9a04-e780e6ce7212" containerName="kube-rbac-proxy"
Jul 28 18:37:33.532532 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.532226    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc5dd1f2-6cfa-45b1-886a-883f1a72c75a" containerName="storage-initializer"
Jul 28 18:37:33.532532 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.532236    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc5dd1f2-6cfa-45b1-886a-883f1a72c75a" containerName="storage-initializer"
Jul 28 18:37:33.532532 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.532248    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="83d8b07b-bd16-4785-9a04-e780e6ce7212" containerName="kserve-container"
Jul 28 18:37:33.532532 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.532308    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc5dd1f2-6cfa-45b1-886a-883f1a72c75a" containerName="storage-initializer"
Jul 28 18:37:33.532532 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.532318    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc5dd1f2-6cfa-45b1-886a-883f1a72c75a" containerName="storage-initializer"
Jul 28 18:37:33.536240 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.536220    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"
Jul 28 18:37:33.539037 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.539003    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Jul 28 18:37:33.539037 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.539032    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:37:33.539230 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.539055    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:37:33.539230 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.539124    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:37:33.539230 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.539142    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rvk9p\""
Jul 28 18:37:33.539230 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.539216    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 28 18:37:33.540093 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.540073    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Jul 28 18:37:33.545655 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.545634    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"]
Jul 28 18:37:33.643003 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.642964    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad2d9042-8847-4060-b43b-a3e56327f84f-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws\" (UID: \"ad2d9042-8847-4060-b43b-a3e56327f84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"
Jul 28 18:37:33.643187 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.643012    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad2d9042-8847-4060-b43b-a3e56327f84f-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws\" (UID: \"ad2d9042-8847-4060-b43b-a3e56327f84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"
Jul 28 18:37:33.643187 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.643072    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad2d9042-8847-4060-b43b-a3e56327f84f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws\" (UID: \"ad2d9042-8847-4060-b43b-a3e56327f84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"
Jul 28 18:37:33.643187 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.643131    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad2d9042-8847-4060-b43b-a3e56327f84f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws\" (UID: \"ad2d9042-8847-4060-b43b-a3e56327f84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"
Jul 28 18:37:33.643187 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.643152    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4bbk\" (UniqueName: \"kubernetes.io/projected/ad2d9042-8847-4060-b43b-a3e56327f84f-kube-api-access-k4bbk\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws\" (UID: \"ad2d9042-8847-4060-b43b-a3e56327f84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"
Jul 28 18:37:33.744313 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.744274    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad2d9042-8847-4060-b43b-a3e56327f84f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws\" (UID: \"ad2d9042-8847-4060-b43b-a3e56327f84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"
Jul 28 18:37:33.744455 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.744332    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad2d9042-8847-4060-b43b-a3e56327f84f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws\" (UID: \"ad2d9042-8847-4060-b43b-a3e56327f84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"
Jul 28 18:37:33.744455 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.744351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4bbk\" (UniqueName: \"kubernetes.io/projected/ad2d9042-8847-4060-b43b-a3e56327f84f-kube-api-access-k4bbk\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws\" (UID: \"ad2d9042-8847-4060-b43b-a3e56327f84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"
Jul 28 18:37:33.744455 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.744386    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad2d9042-8847-4060-b43b-a3e56327f84f-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws\" (UID: \"ad2d9042-8847-4060-b43b-a3e56327f84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"
Jul 28 18:37:33.744455 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.744416    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad2d9042-8847-4060-b43b-a3e56327f84f-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws\" (UID: \"ad2d9042-8847-4060-b43b-a3e56327f84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"
Jul 28 18:37:33.744684 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:37:33.744528    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Jul 28 18:37:33.744684 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:37:33.744595    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad2d9042-8847-4060-b43b-a3e56327f84f-proxy-tls podName:ad2d9042-8847-4060-b43b-a3e56327f84f nodeName:}" failed. No retries permitted until 2026-07-28 18:37:34.244575091 +0000 UTC m=+5302.519191502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ad2d9042-8847-4060-b43b-a3e56327f84f-proxy-tls") pod "isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws" (UID: "ad2d9042-8847-4060-b43b-a3e56327f84f") : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Jul 28 18:37:33.744825 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.744764    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad2d9042-8847-4060-b43b-a3e56327f84f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws\" (UID: \"ad2d9042-8847-4060-b43b-a3e56327f84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"
Jul 28 18:37:33.745103 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.745082    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad2d9042-8847-4060-b43b-a3e56327f84f-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws\" (UID: \"ad2d9042-8847-4060-b43b-a3e56327f84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"
Jul 28 18:37:33.745284 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.745260    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad2d9042-8847-4060-b43b-a3e56327f84f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws\" (UID: \"ad2d9042-8847-4060-b43b-a3e56327f84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"
Jul 28 18:37:33.753548 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.753503    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4bbk\" (UniqueName: \"kubernetes.io/projected/ad2d9042-8847-4060-b43b-a3e56327f84f-kube-api-access-k4bbk\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws\" (UID: \"ad2d9042-8847-4060-b43b-a3e56327f84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"
Jul 28 18:37:33.884729 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:33.884682    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:37:34.247785 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:34.247746    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad2d9042-8847-4060-b43b-a3e56327f84f-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws\" (UID: \"ad2d9042-8847-4060-b43b-a3e56327f84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"
Jul 28 18:37:34.250251 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:34.250216    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad2d9042-8847-4060-b43b-a3e56327f84f-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws\" (UID: \"ad2d9042-8847-4060-b43b-a3e56327f84f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"
Jul 28 18:37:34.289489 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:34.289455    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc5dd1f2-6cfa-45b1-886a-883f1a72c75a" path="/var/lib/kubelet/pods/cc5dd1f2-6cfa-45b1-886a-883f1a72c75a/volumes"
Jul 28 18:37:34.446691 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:34.446649    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"
Jul 28 18:37:34.578522 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:34.578316    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"]
Jul 28 18:37:34.581179 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:37:34.581149    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2d9042_8847_4060_b43b_a3e56327f84f.slice/crio-5292af051281daed1c56c33f2a8233cab3cebf47dc89b1906bfbf09bb0b295ea WatchSource:0}: Error finding container 5292af051281daed1c56c33f2a8233cab3cebf47dc89b1906bfbf09bb0b295ea: Status 404 returned error can't find the container with id 5292af051281daed1c56c33f2a8233cab3cebf47dc89b1906bfbf09bb0b295ea
Jul 28 18:37:34.636987 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:34.636957    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws" event={"ID":"ad2d9042-8847-4060-b43b-a3e56327f84f","Type":"ContainerStarted","Data":"5292af051281daed1c56c33f2a8233cab3cebf47dc89b1906bfbf09bb0b295ea"}
Jul 28 18:37:35.642956 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:35.642908    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws" event={"ID":"ad2d9042-8847-4060-b43b-a3e56327f84f","Type":"ContainerStarted","Data":"2aa352e31041bbd0d9da1cffcae8cb827d1bc70f0858a2f90d9e15e64e581440"}
Jul 28 18:37:36.647505 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:36.647462    2581 generic.go:356] "Generic (PLEG): container finished" podID="ad2d9042-8847-4060-b43b-a3e56327f84f" containerID="2aa352e31041bbd0d9da1cffcae8cb827d1bc70f0858a2f90d9e15e64e581440" exitCode=0
Jul 28 18:37:36.648005 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:36.647546    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws" event={"ID":"ad2d9042-8847-4060-b43b-a3e56327f84f","Type":"ContainerDied","Data":"2aa352e31041bbd0d9da1cffcae8cb827d1bc70f0858a2f90d9e15e64e581440"}
Jul 28 18:37:37.653617 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:37.653582    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws" event={"ID":"ad2d9042-8847-4060-b43b-a3e56327f84f","Type":"ContainerStarted","Data":"5e8b13f9b781d71fe00f34c0dc9ce378f56782ec13905cd792e6f79d811102aa"}
Jul 28 18:37:37.654112 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:37.653642    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws" event={"ID":"ad2d9042-8847-4060-b43b-a3e56327f84f","Type":"ContainerStarted","Data":"269caaebf6d0800d3db0f599f975dfdbc3b7c9db7c4d6268e6896c12f59f2678"}
Jul 28 18:37:37.654112 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:37.653731    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"
Jul 28 18:37:37.675491 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:37.675433    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws" podStartSLOduration=3.610754156 podStartE2EDuration="4.675416233s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.064662077s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:37:37.674343408 +0000 UTC m=+5305.948959840" watchObservedRunningTime="2026-07-28 18:37:37.675416233 +0000 UTC m=+5305.950032664"
Jul 28 18:37:38.657458 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:38.657418    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"
Jul 28 18:37:38.658595 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:38.658566    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws" podUID="ad2d9042-8847-4060-b43b-a3e56327f84f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.72:8080: connect: connection refused"
Jul 28 18:37:39.660928 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:39.660879    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws" podUID="ad2d9042-8847-4060-b43b-a3e56327f84f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.72:8080: connect: connection refused"
Jul 28 18:37:44.667337 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:44.667307    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"
Jul 28 18:37:44.667877 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:44.667848    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws" podUID="ad2d9042-8847-4060-b43b-a3e56327f84f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.72:8080: connect: connection refused"
Jul 28 18:37:54.668805 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:37:54.668761    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws" podUID="ad2d9042-8847-4060-b43b-a3e56327f84f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.72:8080: connect: connection refused"
Jul 28 18:38:04.668713 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:38:04.668669    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws" podUID="ad2d9042-8847-4060-b43b-a3e56327f84f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.72:8080: connect: connection refused"
Jul 28 18:38:14.668333 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:38:14.668284    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws" podUID="ad2d9042-8847-4060-b43b-a3e56327f84f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.72:8080: connect: connection refused"
Jul 28 18:38:24.668053 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:38:24.668013    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws" podUID="ad2d9042-8847-4060-b43b-a3e56327f84f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.72:8080: connect: connection refused"
Jul 28 18:38:34.668061 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:38:34.668020    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws" podUID="ad2d9042-8847-4060-b43b-a3e56327f84f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.72:8080: connect: connection refused"
Jul 28 18:38:44.668467 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:38:44.668422    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws" podUID="ad2d9042-8847-4060-b43b-a3e56327f84f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.72:8080: connect: connection refused"
Jul 28 18:38:54.669169 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:38:54.669138    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"
Jul 28 18:39:03.717227 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:03.717192    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"]
Jul 28 18:39:03.717632 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:03.717444    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws" podUID="ad2d9042-8847-4060-b43b-a3e56327f84f" containerName="kserve-container" containerID="cri-o://269caaebf6d0800d3db0f599f975dfdbc3b7c9db7c4d6268e6896c12f59f2678" gracePeriod=30
Jul 28 18:39:03.717632 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:03.717545    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws" podUID="ad2d9042-8847-4060-b43b-a3e56327f84f" containerName="kube-rbac-proxy" containerID="cri-o://5e8b13f9b781d71fe00f34c0dc9ce378f56782ec13905cd792e6f79d811102aa" gracePeriod=30
Jul 28 18:39:03.738907 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:03.738862    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:39:03.964325 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:03.964284    2581 generic.go:356] "Generic (PLEG): container finished" podID="ad2d9042-8847-4060-b43b-a3e56327f84f" containerID="5e8b13f9b781d71fe00f34c0dc9ce378f56782ec13905cd792e6f79d811102aa" exitCode=2
Jul 28 18:39:03.964503 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:03.964332    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws" event={"ID":"ad2d9042-8847-4060-b43b-a3e56327f84f","Type":"ContainerDied","Data":"5e8b13f9b781d71fe00f34c0dc9ce378f56782ec13905cd792e6f79d811102aa"}
Jul 28 18:39:04.649231 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:04.649195    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:39:04.661888 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:04.661852    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws" podUID="ad2d9042-8847-4060-b43b-a3e56327f84f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.72:8643/healthz\": dial tcp 10.133.0.72:8643: connect: connection refused"
Jul 28 18:39:04.668403 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:04.668373    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws" podUID="ad2d9042-8847-4060-b43b-a3e56327f84f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.72:8080: connect: connection refused"
Jul 28 18:39:04.701873 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:04.701838    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5"]
Jul 28 18:39:04.705593 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:04.705572    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5"
Jul 28 18:39:04.708163 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:04.708140    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Jul 28 18:39:04.708321 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:04.708177    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Jul 28 18:39:04.714889 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:04.714860    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5"]
Jul 28 18:39:04.769344 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:04.769309    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:39:04.831652 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:04.831616    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqfzv\" (UniqueName: \"kubernetes.io/projected/6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c-kube-api-access-gqfzv\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5\" (UID: \"6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5"
Jul 28 18:39:04.831652 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:04.831655    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5\" (UID: \"6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5"
Jul 28 18:39:04.831874 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:04.831681    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5\" (UID: \"6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5"
Jul 28 18:39:04.831874 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:04.831779    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5\" (UID: \"6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5"
Jul 28 18:39:04.933034 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:04.932877    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5\" (UID: \"6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5"
Jul 28 18:39:04.933034 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:04.932963    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqfzv\" (UniqueName: \"kubernetes.io/projected/6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c-kube-api-access-gqfzv\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5\" (UID: \"6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5"
Jul 28 18:39:04.933277 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:04.933082    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5\" (UID: \"6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5"
Jul 28 18:39:04.933277 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:04.933120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5\" (UID: \"6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5"
Jul 28 18:39:04.933560 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:04.933536    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5\" (UID: \"6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5"
Jul 28 18:39:04.933812 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:04.933792    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5\" (UID: \"6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5"
Jul 28 18:39:04.935426 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:04.935405    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5\" (UID: \"6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5"
Jul 28 18:39:04.942928 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:04.942898    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqfzv\" (UniqueName: \"kubernetes.io/projected/6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c-kube-api-access-gqfzv\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5\" (UID: \"6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5"
Jul 28 18:39:05.017075 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:05.017038    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5"
Jul 28 18:39:05.146371 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:05.146335    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5"]
Jul 28 18:39:05.149255 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:39:05.149226    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b04bd16_bab6_4d8c_bcbf_9ed9f954bc4c.slice/crio-6d440b6a72dfbb0a7b056d672013c591f666d7b68c50d164ecfe1eeae3e9f5b2 WatchSource:0}: Error finding container 6d440b6a72dfbb0a7b056d672013c591f666d7b68c50d164ecfe1eeae3e9f5b2: Status 404 returned error can't find the container with id 6d440b6a72dfbb0a7b056d672013c591f666d7b68c50d164ecfe1eeae3e9f5b2
Jul 28 18:39:05.972737 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:05.972697    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5" event={"ID":"6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c","Type":"ContainerStarted","Data":"d2373b1c71cf51202dc1b42fd44f101e2d7d058ccb33ceed0c7f383c7ad7c247"}
Jul 28 18:39:05.973160 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:05.972775    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5" event={"ID":"6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c","Type":"ContainerStarted","Data":"6d440b6a72dfbb0a7b056d672013c591f666d7b68c50d164ecfe1eeae3e9f5b2"}
Jul 28 18:39:08.668451 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:08.668419    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"
Jul 28 18:39:08.765529 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:08.765443    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad2d9042-8847-4060-b43b-a3e56327f84f-cabundle-cert\") pod \"ad2d9042-8847-4060-b43b-a3e56327f84f\" (UID: \"ad2d9042-8847-4060-b43b-a3e56327f84f\") "
Jul 28 18:39:08.765529 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:08.765480    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4bbk\" (UniqueName: \"kubernetes.io/projected/ad2d9042-8847-4060-b43b-a3e56327f84f-kube-api-access-k4bbk\") pod \"ad2d9042-8847-4060-b43b-a3e56327f84f\" (UID: \"ad2d9042-8847-4060-b43b-a3e56327f84f\") "
Jul 28 18:39:08.765755 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:08.765532    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad2d9042-8847-4060-b43b-a3e56327f84f-kube-rbac-proxy-sar-config\") pod \"ad2d9042-8847-4060-b43b-a3e56327f84f\" (UID: \"ad2d9042-8847-4060-b43b-a3e56327f84f\") "
Jul 28 18:39:08.765755 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:08.765565    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad2d9042-8847-4060-b43b-a3e56327f84f-kserve-provision-location\") pod \"ad2d9042-8847-4060-b43b-a3e56327f84f\" (UID: \"ad2d9042-8847-4060-b43b-a3e56327f84f\") "
Jul 28 18:39:08.765755 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:08.765586    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad2d9042-8847-4060-b43b-a3e56327f84f-proxy-tls\") pod \"ad2d9042-8847-4060-b43b-a3e56327f84f\" (UID: \"ad2d9042-8847-4060-b43b-a3e56327f84f\") "
Jul 28 18:39:08.765937 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:08.765793    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad2d9042-8847-4060-b43b-a3e56327f84f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ad2d9042-8847-4060-b43b-a3e56327f84f" (UID: "ad2d9042-8847-4060-b43b-a3e56327f84f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:39:08.765937 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:08.765864    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad2d9042-8847-4060-b43b-a3e56327f84f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ad2d9042-8847-4060-b43b-a3e56327f84f" (UID: "ad2d9042-8847-4060-b43b-a3e56327f84f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:39:08.766085 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:08.765968    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad2d9042-8847-4060-b43b-a3e56327f84f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ad2d9042-8847-4060-b43b-a3e56327f84f" (UID: "ad2d9042-8847-4060-b43b-a3e56327f84f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:39:08.767600 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:08.767577    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad2d9042-8847-4060-b43b-a3e56327f84f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ad2d9042-8847-4060-b43b-a3e56327f84f" (UID: "ad2d9042-8847-4060-b43b-a3e56327f84f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:39:08.767713 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:08.767696    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad2d9042-8847-4060-b43b-a3e56327f84f-kube-api-access-k4bbk" (OuterVolumeSpecName: "kube-api-access-k4bbk") pod "ad2d9042-8847-4060-b43b-a3e56327f84f" (UID: "ad2d9042-8847-4060-b43b-a3e56327f84f"). InnerVolumeSpecName "kube-api-access-k4bbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:39:08.866064 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:08.866023    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad2d9042-8847-4060-b43b-a3e56327f84f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:39:08.866064 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:08.866059    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad2d9042-8847-4060-b43b-a3e56327f84f-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:39:08.866064 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:08.866070    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad2d9042-8847-4060-b43b-a3e56327f84f-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:39:08.866064 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:08.866080    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad2d9042-8847-4060-b43b-a3e56327f84f-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:39:08.866403 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:08.866089    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4bbk\" (UniqueName: \"kubernetes.io/projected/ad2d9042-8847-4060-b43b-a3e56327f84f-kube-api-access-k4bbk\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:39:08.986045 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:08.986008    2581 generic.go:356] "Generic (PLEG): container finished" podID="ad2d9042-8847-4060-b43b-a3e56327f84f" containerID="269caaebf6d0800d3db0f599f975dfdbc3b7c9db7c4d6268e6896c12f59f2678" exitCode=0
Jul 28 18:39:08.986203 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:08.986050    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws" event={"ID":"ad2d9042-8847-4060-b43b-a3e56327f84f","Type":"ContainerDied","Data":"269caaebf6d0800d3db0f599f975dfdbc3b7c9db7c4d6268e6896c12f59f2678"}
Jul 28 18:39:08.986203 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:08.986063    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"
Jul 28 18:39:08.986203 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:08.986106    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws" event={"ID":"ad2d9042-8847-4060-b43b-a3e56327f84f","Type":"ContainerDied","Data":"5292af051281daed1c56c33f2a8233cab3cebf47dc89b1906bfbf09bb0b295ea"}
Jul 28 18:39:08.986203 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:08.986136    2581 scope.go:117] "RemoveContainer" containerID="5e8b13f9b781d71fe00f34c0dc9ce378f56782ec13905cd792e6f79d811102aa"
Jul 28 18:39:08.994255 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:08.994236    2581 scope.go:117] "RemoveContainer" containerID="269caaebf6d0800d3db0f599f975dfdbc3b7c9db7c4d6268e6896c12f59f2678"
Jul 28 18:39:09.001396 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:09.001380    2581 scope.go:117] "RemoveContainer" containerID="2aa352e31041bbd0d9da1cffcae8cb827d1bc70f0858a2f90d9e15e64e581440"
Jul 28 18:39:09.008528 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:09.008508    2581 scope.go:117] "RemoveContainer" containerID="5e8b13f9b781d71fe00f34c0dc9ce378f56782ec13905cd792e6f79d811102aa"
Jul 28 18:39:09.008805 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:39:09.008779    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e8b13f9b781d71fe00f34c0dc9ce378f56782ec13905cd792e6f79d811102aa\": container with ID starting with 5e8b13f9b781d71fe00f34c0dc9ce378f56782ec13905cd792e6f79d811102aa not found: ID does not exist" containerID="5e8b13f9b781d71fe00f34c0dc9ce378f56782ec13905cd792e6f79d811102aa"
Jul 28 18:39:09.008912 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:09.008814    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e8b13f9b781d71fe00f34c0dc9ce378f56782ec13905cd792e6f79d811102aa"} err="failed to get container status \"5e8b13f9b781d71fe00f34c0dc9ce378f56782ec13905cd792e6f79d811102aa\": rpc error: code = NotFound desc = could not find container \"5e8b13f9b781d71fe00f34c0dc9ce378f56782ec13905cd792e6f79d811102aa\": container with ID starting with 5e8b13f9b781d71fe00f34c0dc9ce378f56782ec13905cd792e6f79d811102aa not found: ID does not exist"
Jul 28 18:39:09.008912 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:09.008835    2581 scope.go:117] "RemoveContainer" containerID="269caaebf6d0800d3db0f599f975dfdbc3b7c9db7c4d6268e6896c12f59f2678"
Jul 28 18:39:09.009110 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:39:09.009089    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"269caaebf6d0800d3db0f599f975dfdbc3b7c9db7c4d6268e6896c12f59f2678\": container with ID starting with 269caaebf6d0800d3db0f599f975dfdbc3b7c9db7c4d6268e6896c12f59f2678 not found: ID does not exist" containerID="269caaebf6d0800d3db0f599f975dfdbc3b7c9db7c4d6268e6896c12f59f2678"
Jul 28 18:39:09.009149 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:09.009117    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"269caaebf6d0800d3db0f599f975dfdbc3b7c9db7c4d6268e6896c12f59f2678"} err="failed to get container status \"269caaebf6d0800d3db0f599f975dfdbc3b7c9db7c4d6268e6896c12f59f2678\": rpc error: code = NotFound desc = could not find container \"269caaebf6d0800d3db0f599f975dfdbc3b7c9db7c4d6268e6896c12f59f2678\": container with ID starting with 269caaebf6d0800d3db0f599f975dfdbc3b7c9db7c4d6268e6896c12f59f2678 not found: ID does not exist"
Jul 28 18:39:09.009149 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:09.009133    2581 scope.go:117] "RemoveContainer" containerID="2aa352e31041bbd0d9da1cffcae8cb827d1bc70f0858a2f90d9e15e64e581440"
Jul 28 18:39:09.009371 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:39:09.009352    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aa352e31041bbd0d9da1cffcae8cb827d1bc70f0858a2f90d9e15e64e581440\": container with ID starting with 2aa352e31041bbd0d9da1cffcae8cb827d1bc70f0858a2f90d9e15e64e581440 not found: ID does not exist" containerID="2aa352e31041bbd0d9da1cffcae8cb827d1bc70f0858a2f90d9e15e64e581440"
Jul 28 18:39:09.009429 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:09.009375    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aa352e31041bbd0d9da1cffcae8cb827d1bc70f0858a2f90d9e15e64e581440"} err="failed to get container status \"2aa352e31041bbd0d9da1cffcae8cb827d1bc70f0858a2f90d9e15e64e581440\": rpc error: code = NotFound desc = could not find container \"2aa352e31041bbd0d9da1cffcae8cb827d1bc70f0858a2f90d9e15e64e581440\": container with ID starting with 2aa352e31041bbd0d9da1cffcae8cb827d1bc70f0858a2f90d9e15e64e581440 not found: ID does not exist"
Jul 28 18:39:09.009481 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:09.009465    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"]
Jul 28 18:39:09.014758 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:09.014736    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-8l7ws"]
Jul 28 18:39:10.289866 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:10.289782    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad2d9042-8847-4060-b43b-a3e56327f84f" path="/var/lib/kubelet/pods/ad2d9042-8847-4060-b43b-a3e56327f84f/volumes"
Jul 28 18:39:10.995641 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:10.995610    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5_6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c/storage-initializer/0.log"
Jul 28 18:39:10.995828 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:10.995652    2581 generic.go:356] "Generic (PLEG): container finished" podID="6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c" containerID="d2373b1c71cf51202dc1b42fd44f101e2d7d058ccb33ceed0c7f383c7ad7c247" exitCode=1
Jul 28 18:39:10.995828 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:10.995738    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5" event={"ID":"6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c","Type":"ContainerDied","Data":"d2373b1c71cf51202dc1b42fd44f101e2d7d058ccb33ceed0c7f383c7ad7c247"}
Jul 28 18:39:12.001663 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:12.001630    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5_6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c/storage-initializer/0.log"
Jul 28 18:39:12.002077 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:12.001702    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5" event={"ID":"6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c","Type":"ContainerStarted","Data":"f0dc3b9d8316964e23c35e1c57ce03d0eeb400d158ad0ba382540e6ac767b5ac"}
Jul 28 18:39:12.517163 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:12.517135    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5_6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c/storage-initializer/0.log"
Jul 28 18:39:12.527894 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:12.527865    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 18:39:12.528100 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:12.528073    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5_6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c/storage-initializer/0.log"
Jul 28 18:39:12.541034 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:12.541009    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 18:39:14.676585 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:14.676549    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5"]
Jul 28 18:39:14.677037 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:14.676819    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5" podUID="6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c" containerName="storage-initializer" containerID="cri-o://f0dc3b9d8316964e23c35e1c57ce03d0eeb400d158ad0ba382540e6ac767b5ac" gracePeriod=30
Jul 28 18:39:14.690516 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:14.690488    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:39:15.748142 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:15.748101    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:39:15.778696 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:15.778656    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"]
Jul 28 18:39:15.778978 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:15.778962    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad2d9042-8847-4060-b43b-a3e56327f84f" containerName="storage-initializer"
Jul 28 18:39:15.779029 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:15.778981    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad2d9042-8847-4060-b43b-a3e56327f84f" containerName="storage-initializer"
Jul 28 18:39:15.779029 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:15.778994    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad2d9042-8847-4060-b43b-a3e56327f84f" containerName="kube-rbac-proxy"
Jul 28 18:39:15.779029 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:15.779000    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad2d9042-8847-4060-b43b-a3e56327f84f" containerName="kube-rbac-proxy"
Jul 28 18:39:15.779029 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:15.779007    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad2d9042-8847-4060-b43b-a3e56327f84f" containerName="kserve-container"
Jul 28 18:39:15.779029 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:15.779014    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad2d9042-8847-4060-b43b-a3e56327f84f" containerName="kserve-container"
Jul 28 18:39:15.779187 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:15.779069    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad2d9042-8847-4060-b43b-a3e56327f84f" containerName="kube-rbac-proxy"
Jul 28 18:39:15.779187 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:15.779078    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad2d9042-8847-4060-b43b-a3e56327f84f" containerName="kserve-container"
Jul 28 18:39:15.782926 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:15.782904    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"
Jul 28 18:39:15.785600 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:15.785572    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:39:15.785765 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:15.785600    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Jul 28 18:39:15.785765 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:15.785656    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Jul 28 18:39:15.792179 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:15.792147    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"]
Jul 28 18:39:15.813001 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:15.812966    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:39:15.918721 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:15.918682    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ac4efe6-3f27-405d-9da4-acf6ac07601f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj\" (UID: \"5ac4efe6-3f27-405d-9da4-acf6ac07601f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"
Jul 28 18:39:15.918721 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:15.918725    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thwnh\" (UniqueName: \"kubernetes.io/projected/5ac4efe6-3f27-405d-9da4-acf6ac07601f-kube-api-access-thwnh\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj\" (UID: \"5ac4efe6-3f27-405d-9da4-acf6ac07601f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"
Jul 28 18:39:15.918967 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:15.918752    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ac4efe6-3f27-405d-9da4-acf6ac07601f-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj\" (UID: \"5ac4efe6-3f27-405d-9da4-acf6ac07601f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"
Jul 28 18:39:15.918967 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:15.918825    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ac4efe6-3f27-405d-9da4-acf6ac07601f-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj\" (UID: \"5ac4efe6-3f27-405d-9da4-acf6ac07601f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"
Jul 28 18:39:15.918967 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:15.918874    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ac4efe6-3f27-405d-9da4-acf6ac07601f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj\" (UID: \"5ac4efe6-3f27-405d-9da4-acf6ac07601f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"
Jul 28 18:39:16.020188 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:16.020100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ac4efe6-3f27-405d-9da4-acf6ac07601f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj\" (UID: \"5ac4efe6-3f27-405d-9da4-acf6ac07601f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"
Jul 28 18:39:16.020188 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:16.020169    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ac4efe6-3f27-405d-9da4-acf6ac07601f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj\" (UID: \"5ac4efe6-3f27-405d-9da4-acf6ac07601f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"
Jul 28 18:39:16.020420 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:16.020195    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thwnh\" (UniqueName: \"kubernetes.io/projected/5ac4efe6-3f27-405d-9da4-acf6ac07601f-kube-api-access-thwnh\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj\" (UID: \"5ac4efe6-3f27-405d-9da4-acf6ac07601f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"
Jul 28 18:39:16.020420 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:16.020216    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ac4efe6-3f27-405d-9da4-acf6ac07601f-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj\" (UID: \"5ac4efe6-3f27-405d-9da4-acf6ac07601f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"
Jul 28 18:39:16.020420 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:16.020245    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ac4efe6-3f27-405d-9da4-acf6ac07601f-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj\" (UID: \"5ac4efe6-3f27-405d-9da4-acf6ac07601f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"
Jul 28 18:39:16.020602 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:16.020575    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ac4efe6-3f27-405d-9da4-acf6ac07601f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj\" (UID: \"5ac4efe6-3f27-405d-9da4-acf6ac07601f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"
Jul 28 18:39:16.020968 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:16.020925    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ac4efe6-3f27-405d-9da4-acf6ac07601f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj\" (UID: \"5ac4efe6-3f27-405d-9da4-acf6ac07601f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"
Jul 28 18:39:16.020968 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:16.020927    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ac4efe6-3f27-405d-9da4-acf6ac07601f-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj\" (UID: \"5ac4efe6-3f27-405d-9da4-acf6ac07601f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"
Jul 28 18:39:16.022757 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:16.022726    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ac4efe6-3f27-405d-9da4-acf6ac07601f-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj\" (UID: \"5ac4efe6-3f27-405d-9da4-acf6ac07601f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"
Jul 28 18:39:16.028890 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:16.028857    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thwnh\" (UniqueName: \"kubernetes.io/projected/5ac4efe6-3f27-405d-9da4-acf6ac07601f-kube-api-access-thwnh\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj\" (UID: \"5ac4efe6-3f27-405d-9da4-acf6ac07601f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"
Jul 28 18:39:16.094062 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:16.094019    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"
Jul 28 18:39:16.226306 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:16.226267    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"]
Jul 28 18:39:16.229146 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:39:16.229111    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac4efe6_3f27_405d_9da4_acf6ac07601f.slice/crio-f397f3fd02c52557968264b1efcb30f2cc9189e7bdf1f3f923348d4ad5a0cd96 WatchSource:0}: Error finding container f397f3fd02c52557968264b1efcb30f2cc9189e7bdf1f3f923348d4ad5a0cd96: Status 404 returned error can't find the container with id f397f3fd02c52557968264b1efcb30f2cc9189e7bdf1f3f923348d4ad5a0cd96
Jul 28 18:39:16.606870 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:16.606846    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5_6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c/storage-initializer/1.log"
Jul 28 18:39:16.607337 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:16.607317    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5_6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c/storage-initializer/0.log"
Jul 28 18:39:16.609763 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:16.609743    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5"
Jul 28 18:39:16.725670 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:16.725630    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c-kube-rbac-proxy-sar-config\") pod \"6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c\" (UID: \"6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c\") "
Jul 28 18:39:16.725856 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:16.725693    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c-kserve-provision-location\") pod \"6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c\" (UID: \"6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c\") "
Jul 28 18:39:16.725856 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:16.725724    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqfzv\" (UniqueName: \"kubernetes.io/projected/6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c-kube-api-access-gqfzv\") pod \"6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c\" (UID: \"6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c\") "
Jul 28 18:39:16.725856 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:16.725742    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c-proxy-tls\") pod \"6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c\" (UID: \"6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c\") "
Jul 28 18:39:16.726102 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:16.726072    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c" (UID: "6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:39:16.726168 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:16.726075    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c" (UID: "6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:39:16.727914 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:16.727892    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c-kube-api-access-gqfzv" (OuterVolumeSpecName: "kube-api-access-gqfzv") pod "6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c" (UID: "6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c"). InnerVolumeSpecName "kube-api-access-gqfzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:39:16.728004 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:16.727907    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c" (UID: "6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:39:16.827038 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:16.826928    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:39:16.827038 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:16.826975    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqfzv\" (UniqueName: \"kubernetes.io/projected/6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c-kube-api-access-gqfzv\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:39:16.827038 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:16.826988    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:39:16.827038 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:16.827000    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:39:17.020244 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:17.020202    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj" event={"ID":"5ac4efe6-3f27-405d-9da4-acf6ac07601f","Type":"ContainerStarted","Data":"9e121860d4e8dd4a5e3dd432b095e650a17f6b8e41235db039ecd462439aba56"}
Jul 28 18:39:17.020485 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:17.020283    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj" event={"ID":"5ac4efe6-3f27-405d-9da4-acf6ac07601f","Type":"ContainerStarted","Data":"f397f3fd02c52557968264b1efcb30f2cc9189e7bdf1f3f923348d4ad5a0cd96"}
Jul 28 18:39:17.021652 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:17.021633    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5_6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c/storage-initializer/1.log"
Jul 28 18:39:17.022075 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:17.022059    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5_6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c/storage-initializer/0.log"
Jul 28 18:39:17.022155 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:17.022094    2581 generic.go:356] "Generic (PLEG): container finished" podID="6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c" containerID="f0dc3b9d8316964e23c35e1c57ce03d0eeb400d158ad0ba382540e6ac767b5ac" exitCode=1
Jul 28 18:39:17.022155 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:17.022144    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5" event={"ID":"6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c","Type":"ContainerDied","Data":"f0dc3b9d8316964e23c35e1c57ce03d0eeb400d158ad0ba382540e6ac767b5ac"}
Jul 28 18:39:17.022155 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:17.022143    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5"
Jul 28 18:39:17.022307 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:17.022193    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5" event={"ID":"6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c","Type":"ContainerDied","Data":"6d440b6a72dfbb0a7b056d672013c591f666d7b68c50d164ecfe1eeae3e9f5b2"}
Jul 28 18:39:17.022307 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:17.022223    2581 scope.go:117] "RemoveContainer" containerID="f0dc3b9d8316964e23c35e1c57ce03d0eeb400d158ad0ba382540e6ac767b5ac"
Jul 28 18:39:17.039685 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:17.039655    2581 scope.go:117] "RemoveContainer" containerID="d2373b1c71cf51202dc1b42fd44f101e2d7d058ccb33ceed0c7f383c7ad7c247"
Jul 28 18:39:17.047347 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:17.047325    2581 scope.go:117] "RemoveContainer" containerID="f0dc3b9d8316964e23c35e1c57ce03d0eeb400d158ad0ba382540e6ac767b5ac"
Jul 28 18:39:17.047643 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:39:17.047621    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0dc3b9d8316964e23c35e1c57ce03d0eeb400d158ad0ba382540e6ac767b5ac\": container with ID starting with f0dc3b9d8316964e23c35e1c57ce03d0eeb400d158ad0ba382540e6ac767b5ac not found: ID does not exist" containerID="f0dc3b9d8316964e23c35e1c57ce03d0eeb400d158ad0ba382540e6ac767b5ac"
Jul 28 18:39:17.047699 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:17.047651    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0dc3b9d8316964e23c35e1c57ce03d0eeb400d158ad0ba382540e6ac767b5ac"} err="failed to get container status \"f0dc3b9d8316964e23c35e1c57ce03d0eeb400d158ad0ba382540e6ac767b5ac\": rpc error: code = NotFound desc = could not find container \"f0dc3b9d8316964e23c35e1c57ce03d0eeb400d158ad0ba382540e6ac767b5ac\": container with ID starting with f0dc3b9d8316964e23c35e1c57ce03d0eeb400d158ad0ba382540e6ac767b5ac not found: ID does not exist"
Jul 28 18:39:17.047699 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:17.047670    2581 scope.go:117] "RemoveContainer" containerID="d2373b1c71cf51202dc1b42fd44f101e2d7d058ccb33ceed0c7f383c7ad7c247"
Jul 28 18:39:17.047926 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:39:17.047903    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2373b1c71cf51202dc1b42fd44f101e2d7d058ccb33ceed0c7f383c7ad7c247\": container with ID starting with d2373b1c71cf51202dc1b42fd44f101e2d7d058ccb33ceed0c7f383c7ad7c247 not found: ID does not exist" containerID="d2373b1c71cf51202dc1b42fd44f101e2d7d058ccb33ceed0c7f383c7ad7c247"
Jul 28 18:39:17.048012 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:17.047934    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2373b1c71cf51202dc1b42fd44f101e2d7d058ccb33ceed0c7f383c7ad7c247"} err="failed to get container status \"d2373b1c71cf51202dc1b42fd44f101e2d7d058ccb33ceed0c7f383c7ad7c247\": rpc error: code = NotFound desc = could not find container \"d2373b1c71cf51202dc1b42fd44f101e2d7d058ccb33ceed0c7f383c7ad7c247\": container with ID starting with d2373b1c71cf51202dc1b42fd44f101e2d7d058ccb33ceed0c7f383c7ad7c247 not found: ID does not exist"
Jul 28 18:39:17.074154 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:17.074121    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5"]
Jul 28 18:39:17.082391 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:17.082307    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-77d7cc7885-dsdq5"]
Jul 28 18:39:18.028178 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:18.028140    2581 generic.go:356] "Generic (PLEG): container finished" podID="5ac4efe6-3f27-405d-9da4-acf6ac07601f" containerID="9e121860d4e8dd4a5e3dd432b095e650a17f6b8e41235db039ecd462439aba56" exitCode=0
Jul 28 18:39:18.028620 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:18.028187    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj" event={"ID":"5ac4efe6-3f27-405d-9da4-acf6ac07601f","Type":"ContainerDied","Data":"9e121860d4e8dd4a5e3dd432b095e650a17f6b8e41235db039ecd462439aba56"}
Jul 28 18:39:18.290009 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:18.289921    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c" path="/var/lib/kubelet/pods/6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c/volumes"
Jul 28 18:39:19.033804 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:19.033768    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj" event={"ID":"5ac4efe6-3f27-405d-9da4-acf6ac07601f","Type":"ContainerStarted","Data":"e9ef22f46e54204d45bc400b156058095a805f3cebe01d4c40ca8b19965ba6d9"}
Jul 28 18:39:19.034286 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:19.033833    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj" event={"ID":"5ac4efe6-3f27-405d-9da4-acf6ac07601f","Type":"ContainerStarted","Data":"eb1813c85d4830e465ae5669b92bedb9c91cf7c194cd155ddbb108e22dc5364e"}
Jul 28 18:39:19.034286 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:19.033862    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"
Jul 28 18:39:19.034286 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:19.033909    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"
Jul 28 18:39:19.035410 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:19.035384    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj" podUID="5ac4efe6-3f27-405d-9da4-acf6ac07601f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.74:8080: connect: connection refused"
Jul 28 18:39:19.057555 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:19.057504    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj" podStartSLOduration=2.9763497880000003 podStartE2EDuration="4.057488042s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.081138254s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:39:19.056814417 +0000 UTC m=+5407.331430849" watchObservedRunningTime="2026-07-28 18:39:19.057488042 +0000 UTC m=+5407.332104473"
Jul 28 18:39:20.037635 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:20.037588    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj" podUID="5ac4efe6-3f27-405d-9da4-acf6ac07601f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.74:8080: connect: connection refused"
Jul 28 18:39:25.043017 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:25.042984    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"
Jul 28 18:39:25.043597 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:25.043567    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj" podUID="5ac4efe6-3f27-405d-9da4-acf6ac07601f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.74:8080: connect: connection refused"
Jul 28 18:39:35.043737 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:35.043689    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj" podUID="5ac4efe6-3f27-405d-9da4-acf6ac07601f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.74:8080: connect: connection refused"
Jul 28 18:39:45.044359 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:45.044314    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj" podUID="5ac4efe6-3f27-405d-9da4-acf6ac07601f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.74:8080: connect: connection refused"
Jul 28 18:39:55.044488 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:39:55.044440    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj" podUID="5ac4efe6-3f27-405d-9da4-acf6ac07601f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.74:8080: connect: connection refused"
Jul 28 18:40:05.043655 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:05.043557    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj" podUID="5ac4efe6-3f27-405d-9da4-acf6ac07601f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.74:8080: connect: connection refused"
Jul 28 18:40:15.044521 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:15.044480    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj" podUID="5ac4efe6-3f27-405d-9da4-acf6ac07601f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.74:8080: connect: connection refused"
Jul 28 18:40:25.043513 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:25.043464    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj" podUID="5ac4efe6-3f27-405d-9da4-acf6ac07601f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.74:8080: connect: connection refused"
Jul 28 18:40:31.285760 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:31.285721    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj" podUID="5ac4efe6-3f27-405d-9da4-acf6ac07601f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.74:8080: connect: connection refused"
Jul 28 18:40:41.287159 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:41.287073    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"
Jul 28 18:40:45.852557 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:45.852524    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"]
Jul 28 18:40:45.853005 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:45.852753    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj" podUID="5ac4efe6-3f27-405d-9da4-acf6ac07601f" containerName="kserve-container" containerID="cri-o://eb1813c85d4830e465ae5669b92bedb9c91cf7c194cd155ddbb108e22dc5364e" gracePeriod=30
Jul 28 18:40:45.853005 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:45.852837    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj" podUID="5ac4efe6-3f27-405d-9da4-acf6ac07601f" containerName="kube-rbac-proxy" containerID="cri-o://e9ef22f46e54204d45bc400b156058095a805f3cebe01d4c40ca8b19965ba6d9" gracePeriod=30
Jul 28 18:40:45.902127 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:45.902074    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:40:46.345357 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:46.345317    2581 generic.go:356] "Generic (PLEG): container finished" podID="5ac4efe6-3f27-405d-9da4-acf6ac07601f" containerID="e9ef22f46e54204d45bc400b156058095a805f3cebe01d4c40ca8b19965ba6d9" exitCode=2
Jul 28 18:40:46.345542 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:46.345370    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj" event={"ID":"5ac4efe6-3f27-405d-9da4-acf6ac07601f","Type":"ContainerDied","Data":"e9ef22f46e54204d45bc400b156058095a805f3cebe01d4c40ca8b19965ba6d9"}
Jul 28 18:40:47.001206 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:47.001168    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45"]
Jul 28 18:40:47.001601 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:47.001433    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c" containerName="storage-initializer"
Jul 28 18:40:47.001601 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:47.001444    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c" containerName="storage-initializer"
Jul 28 18:40:47.001601 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:47.001454    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c" containerName="storage-initializer"
Jul 28 18:40:47.001601 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:47.001460    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c" containerName="storage-initializer"
Jul 28 18:40:47.001601 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:47.001510    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c" containerName="storage-initializer"
Jul 28 18:40:47.001601 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:47.001517    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b04bd16-bab6-4d8c-bcbf-9ed9f954bc4c" containerName="storage-initializer"
Jul 28 18:40:47.004554 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:47.004534    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45"
Jul 28 18:40:47.007382 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:47.007359    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Jul 28 18:40:47.007498 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:47.007360    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Jul 28 18:40:47.018440 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:47.018412    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45"]
Jul 28 18:40:47.110966 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:47.110897    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9836ada4-8a3e-40fd-b74f-d6270d1f1328-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45\" (UID: \"9836ada4-8a3e-40fd-b74f-d6270d1f1328\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45"
Jul 28 18:40:47.110966 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:47.110961    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmgw7\" (UniqueName: \"kubernetes.io/projected/9836ada4-8a3e-40fd-b74f-d6270d1f1328-kube-api-access-mmgw7\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45\" (UID: \"9836ada4-8a3e-40fd-b74f-d6270d1f1328\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45"
Jul 28 18:40:47.111208 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:47.111079    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9836ada4-8a3e-40fd-b74f-d6270d1f1328-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45\" (UID: \"9836ada4-8a3e-40fd-b74f-d6270d1f1328\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45"
Jul 28 18:40:47.111208 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:47.111134    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9836ada4-8a3e-40fd-b74f-d6270d1f1328-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45\" (UID: \"9836ada4-8a3e-40fd-b74f-d6270d1f1328\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45"
Jul 28 18:40:47.212095 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:47.212055    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9836ada4-8a3e-40fd-b74f-d6270d1f1328-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45\" (UID: \"9836ada4-8a3e-40fd-b74f-d6270d1f1328\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45"
Jul 28 18:40:47.212095 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:47.212099    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmgw7\" (UniqueName: \"kubernetes.io/projected/9836ada4-8a3e-40fd-b74f-d6270d1f1328-kube-api-access-mmgw7\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45\" (UID: \"9836ada4-8a3e-40fd-b74f-d6270d1f1328\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45"
Jul 28 18:40:47.212374 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:47.212144    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9836ada4-8a3e-40fd-b74f-d6270d1f1328-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45\" (UID: \"9836ada4-8a3e-40fd-b74f-d6270d1f1328\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45"
Jul 28 18:40:47.212374 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:47.212168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9836ada4-8a3e-40fd-b74f-d6270d1f1328-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45\" (UID: \"9836ada4-8a3e-40fd-b74f-d6270d1f1328\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45"
Jul 28 18:40:47.212631 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:47.212607    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9836ada4-8a3e-40fd-b74f-d6270d1f1328-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45\" (UID: \"9836ada4-8a3e-40fd-b74f-d6270d1f1328\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45"
Jul 28 18:40:47.212769 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:47.212740    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9836ada4-8a3e-40fd-b74f-d6270d1f1328-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45\" (UID: \"9836ada4-8a3e-40fd-b74f-d6270d1f1328\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45"
Jul 28 18:40:47.214638 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:47.214619    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9836ada4-8a3e-40fd-b74f-d6270d1f1328-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45\" (UID: \"9836ada4-8a3e-40fd-b74f-d6270d1f1328\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45"
Jul 28 18:40:47.221558 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:47.221534    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmgw7\" (UniqueName: \"kubernetes.io/projected/9836ada4-8a3e-40fd-b74f-d6270d1f1328-kube-api-access-mmgw7\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45\" (UID: \"9836ada4-8a3e-40fd-b74f-d6270d1f1328\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45"
Jul 28 18:40:47.314221 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:47.314126    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45"
Jul 28 18:40:47.445292 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:47.445261    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45"]
Jul 28 18:40:47.448467 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:40:47.448434    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9836ada4_8a3e_40fd_b74f_d6270d1f1328.slice/crio-5e6b77681f43d64e96b26bc21042da90ff1c406030d1608dd7a4cb3993f6f4a8 WatchSource:0}: Error finding container 5e6b77681f43d64e96b26bc21042da90ff1c406030d1608dd7a4cb3993f6f4a8: Status 404 returned error can't find the container with id 5e6b77681f43d64e96b26bc21042da90ff1c406030d1608dd7a4cb3993f6f4a8
Jul 28 18:40:47.525217 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:47.525162    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:40:48.353562 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:48.353526    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45" event={"ID":"9836ada4-8a3e-40fd-b74f-d6270d1f1328","Type":"ContainerStarted","Data":"5368ec11f5fc5901b082f0c995cefdaf2ef8b7ab49e2cd7edb224982e448334f"}
Jul 28 18:40:48.353994 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:48.353584    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45" event={"ID":"9836ada4-8a3e-40fd-b74f-d6270d1f1328","Type":"ContainerStarted","Data":"5e6b77681f43d64e96b26bc21042da90ff1c406030d1608dd7a4cb3993f6f4a8"}
Jul 28 18:40:50.038180 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:50.038138    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj" podUID="5ac4efe6-3f27-405d-9da4-acf6ac07601f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.74:8643/healthz\": dial tcp 10.133.0.74:8643: connect: connection refused"
Jul 28 18:40:50.897076 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:50.897048    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"
Jul 28 18:40:51.045446 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.045350    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ac4efe6-3f27-405d-9da4-acf6ac07601f-cabundle-cert\") pod \"5ac4efe6-3f27-405d-9da4-acf6ac07601f\" (UID: \"5ac4efe6-3f27-405d-9da4-acf6ac07601f\") "
Jul 28 18:40:51.045446 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.045396    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ac4efe6-3f27-405d-9da4-acf6ac07601f-proxy-tls\") pod \"5ac4efe6-3f27-405d-9da4-acf6ac07601f\" (UID: \"5ac4efe6-3f27-405d-9da4-acf6ac07601f\") "
Jul 28 18:40:51.045446 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.045436    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thwnh\" (UniqueName: \"kubernetes.io/projected/5ac4efe6-3f27-405d-9da4-acf6ac07601f-kube-api-access-thwnh\") pod \"5ac4efe6-3f27-405d-9da4-acf6ac07601f\" (UID: \"5ac4efe6-3f27-405d-9da4-acf6ac07601f\") "
Jul 28 18:40:51.046023 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.045462    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ac4efe6-3f27-405d-9da4-acf6ac07601f-kube-rbac-proxy-sar-config\") pod \"5ac4efe6-3f27-405d-9da4-acf6ac07601f\" (UID: \"5ac4efe6-3f27-405d-9da4-acf6ac07601f\") "
Jul 28 18:40:51.046023 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.045493    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ac4efe6-3f27-405d-9da4-acf6ac07601f-kserve-provision-location\") pod \"5ac4efe6-3f27-405d-9da4-acf6ac07601f\" (UID: \"5ac4efe6-3f27-405d-9da4-acf6ac07601f\") "
Jul 28 18:40:51.046023 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.045857    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ac4efe6-3f27-405d-9da4-acf6ac07601f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5ac4efe6-3f27-405d-9da4-acf6ac07601f" (UID: "5ac4efe6-3f27-405d-9da4-acf6ac07601f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:40:51.046023 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.045892    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ac4efe6-3f27-405d-9da4-acf6ac07601f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5ac4efe6-3f27-405d-9da4-acf6ac07601f" (UID: "5ac4efe6-3f27-405d-9da4-acf6ac07601f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:40:51.046023 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.045899    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac4efe6-3f27-405d-9da4-acf6ac07601f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5ac4efe6-3f27-405d-9da4-acf6ac07601f" (UID: "5ac4efe6-3f27-405d-9da4-acf6ac07601f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:40:51.047572 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.047547    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac4efe6-3f27-405d-9da4-acf6ac07601f-kube-api-access-thwnh" (OuterVolumeSpecName: "kube-api-access-thwnh") pod "5ac4efe6-3f27-405d-9da4-acf6ac07601f" (UID: "5ac4efe6-3f27-405d-9da4-acf6ac07601f"). InnerVolumeSpecName "kube-api-access-thwnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:40:51.047679 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.047586    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ac4efe6-3f27-405d-9da4-acf6ac07601f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5ac4efe6-3f27-405d-9da4-acf6ac07601f" (UID: "5ac4efe6-3f27-405d-9da4-acf6ac07601f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:40:51.146521 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.146482    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ac4efe6-3f27-405d-9da4-acf6ac07601f-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:40:51.146521 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.146513    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ac4efe6-3f27-405d-9da4-acf6ac07601f-cabundle-cert\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:40:51.146521 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.146524    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ac4efe6-3f27-405d-9da4-acf6ac07601f-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:40:51.146775 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.146533    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thwnh\" (UniqueName: \"kubernetes.io/projected/5ac4efe6-3f27-405d-9da4-acf6ac07601f-kube-api-access-thwnh\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:40:51.146775 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.146542    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ac4efe6-3f27-405d-9da4-acf6ac07601f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:40:51.366365 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.366259    2581 generic.go:356] "Generic (PLEG): container finished" podID="5ac4efe6-3f27-405d-9da4-acf6ac07601f" containerID="eb1813c85d4830e465ae5669b92bedb9c91cf7c194cd155ddbb108e22dc5364e" exitCode=0
Jul 28 18:40:51.366365 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.366319    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"
Jul 28 18:40:51.366365 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.366339    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj" event={"ID":"5ac4efe6-3f27-405d-9da4-acf6ac07601f","Type":"ContainerDied","Data":"eb1813c85d4830e465ae5669b92bedb9c91cf7c194cd155ddbb108e22dc5364e"}
Jul 28 18:40:51.366655 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.366407    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj" event={"ID":"5ac4efe6-3f27-405d-9da4-acf6ac07601f","Type":"ContainerDied","Data":"f397f3fd02c52557968264b1efcb30f2cc9189e7bdf1f3f923348d4ad5a0cd96"}
Jul 28 18:40:51.366655 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.366435    2581 scope.go:117] "RemoveContainer" containerID="e9ef22f46e54204d45bc400b156058095a805f3cebe01d4c40ca8b19965ba6d9"
Jul 28 18:40:51.381519 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.381496    2581 scope.go:117] "RemoveContainer" containerID="eb1813c85d4830e465ae5669b92bedb9c91cf7c194cd155ddbb108e22dc5364e"
Jul 28 18:40:51.389071 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.389052    2581 scope.go:117] "RemoveContainer" containerID="9e121860d4e8dd4a5e3dd432b095e650a17f6b8e41235db039ecd462439aba56"
Jul 28 18:40:51.392432 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.392410    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"]
Jul 28 18:40:51.396314 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.396291    2581 scope.go:117] "RemoveContainer" containerID="e9ef22f46e54204d45bc400b156058095a805f3cebe01d4c40ca8b19965ba6d9"
Jul 28 18:40:51.396661 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:40:51.396635    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9ef22f46e54204d45bc400b156058095a805f3cebe01d4c40ca8b19965ba6d9\": container with ID starting with e9ef22f46e54204d45bc400b156058095a805f3cebe01d4c40ca8b19965ba6d9 not found: ID does not exist" containerID="e9ef22f46e54204d45bc400b156058095a805f3cebe01d4c40ca8b19965ba6d9"
Jul 28 18:40:51.396758 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.396670    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ef22f46e54204d45bc400b156058095a805f3cebe01d4c40ca8b19965ba6d9"} err="failed to get container status \"e9ef22f46e54204d45bc400b156058095a805f3cebe01d4c40ca8b19965ba6d9\": rpc error: code = NotFound desc = could not find container \"e9ef22f46e54204d45bc400b156058095a805f3cebe01d4c40ca8b19965ba6d9\": container with ID starting with e9ef22f46e54204d45bc400b156058095a805f3cebe01d4c40ca8b19965ba6d9 not found: ID does not exist"
Jul 28 18:40:51.396758 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.396688    2581 scope.go:117] "RemoveContainer" containerID="eb1813c85d4830e465ae5669b92bedb9c91cf7c194cd155ddbb108e22dc5364e"
Jul 28 18:40:51.397015 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:40:51.396987    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb1813c85d4830e465ae5669b92bedb9c91cf7c194cd155ddbb108e22dc5364e\": container with ID starting with eb1813c85d4830e465ae5669b92bedb9c91cf7c194cd155ddbb108e22dc5364e not found: ID does not exist" containerID="eb1813c85d4830e465ae5669b92bedb9c91cf7c194cd155ddbb108e22dc5364e"
Jul 28 18:40:51.397105 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.397023    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb1813c85d4830e465ae5669b92bedb9c91cf7c194cd155ddbb108e22dc5364e"} err="failed to get container status \"eb1813c85d4830e465ae5669b92bedb9c91cf7c194cd155ddbb108e22dc5364e\": rpc error: code = NotFound desc = could not find container \"eb1813c85d4830e465ae5669b92bedb9c91cf7c194cd155ddbb108e22dc5364e\": container with ID starting with eb1813c85d4830e465ae5669b92bedb9c91cf7c194cd155ddbb108e22dc5364e not found: ID does not exist"
Jul 28 18:40:51.397105 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.397044    2581 scope.go:117] "RemoveContainer" containerID="9e121860d4e8dd4a5e3dd432b095e650a17f6b8e41235db039ecd462439aba56"
Jul 28 18:40:51.397287 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:40:51.397269    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e121860d4e8dd4a5e3dd432b095e650a17f6b8e41235db039ecd462439aba56\": container with ID starting with 9e121860d4e8dd4a5e3dd432b095e650a17f6b8e41235db039ecd462439aba56 not found: ID does not exist" containerID="9e121860d4e8dd4a5e3dd432b095e650a17f6b8e41235db039ecd462439aba56"
Jul 28 18:40:51.397345 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.397291    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e121860d4e8dd4a5e3dd432b095e650a17f6b8e41235db039ecd462439aba56"} err="failed to get container status \"9e121860d4e8dd4a5e3dd432b095e650a17f6b8e41235db039ecd462439aba56\": rpc error: code = NotFound desc = could not find container \"9e121860d4e8dd4a5e3dd432b095e650a17f6b8e41235db039ecd462439aba56\": container with ID starting with 9e121860d4e8dd4a5e3dd432b095e650a17f6b8e41235db039ecd462439aba56 not found: ID does not exist"
Jul 28 18:40:51.397750 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:51.397728    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-m24zj"]
Jul 28 18:40:52.290690 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:52.290656    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ac4efe6-3f27-405d-9da4-acf6ac07601f" path="/var/lib/kubelet/pods/5ac4efe6-3f27-405d-9da4-acf6ac07601f/volumes"
Jul 28 18:40:52.371718 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:52.371690    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45_9836ada4-8a3e-40fd-b74f-d6270d1f1328/storage-initializer/0.log"
Jul 28 18:40:52.371890 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:52.371728    2581 generic.go:356] "Generic (PLEG): container finished" podID="9836ada4-8a3e-40fd-b74f-d6270d1f1328" containerID="5368ec11f5fc5901b082f0c995cefdaf2ef8b7ab49e2cd7edb224982e448334f" exitCode=1
Jul 28 18:40:52.371890 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:52.371811    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45" event={"ID":"9836ada4-8a3e-40fd-b74f-d6270d1f1328","Type":"ContainerDied","Data":"5368ec11f5fc5901b082f0c995cefdaf2ef8b7ab49e2cd7edb224982e448334f"}
Jul 28 18:40:53.379045 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:53.379020    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45_9836ada4-8a3e-40fd-b74f-d6270d1f1328/storage-initializer/0.log"
Jul 28 18:40:53.379462 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:53.379113    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45" event={"ID":"9836ada4-8a3e-40fd-b74f-d6270d1f1328","Type":"ContainerStarted","Data":"7792ef646698a9121df3fe38bdabca03dc81b6c1172a74a582a92a8a57297d03"}
Jul 28 18:40:54.384796 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:54.384764    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45_9836ada4-8a3e-40fd-b74f-d6270d1f1328/storage-initializer/1.log"
Jul 28 18:40:54.385269 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:54.385176    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45_9836ada4-8a3e-40fd-b74f-d6270d1f1328/storage-initializer/0.log"
Jul 28 18:40:54.385269 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:54.385212    2581 generic.go:356] "Generic (PLEG): container finished" podID="9836ada4-8a3e-40fd-b74f-d6270d1f1328" containerID="7792ef646698a9121df3fe38bdabca03dc81b6c1172a74a582a92a8a57297d03" exitCode=1
Jul 28 18:40:54.385269 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:54.385249    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45" event={"ID":"9836ada4-8a3e-40fd-b74f-d6270d1f1328","Type":"ContainerDied","Data":"7792ef646698a9121df3fe38bdabca03dc81b6c1172a74a582a92a8a57297d03"}
Jul 28 18:40:54.385425 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:54.385307    2581 scope.go:117] "RemoveContainer" containerID="5368ec11f5fc5901b082f0c995cefdaf2ef8b7ab49e2cd7edb224982e448334f"
Jul 28 18:40:54.385704 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:54.385686    2581 scope.go:117] "RemoveContainer" containerID="5368ec11f5fc5901b082f0c995cefdaf2ef8b7ab49e2cd7edb224982e448334f"
Jul 28 18:40:54.396224 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:40:54.396192    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45_kserve-ci-e2e-test(9836ada4-8a3e-40fd-b74f-d6270d1f1328)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45" podUID="9836ada4-8a3e-40fd-b74f-d6270d1f1328"
Jul 28 18:40:54.396331 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:40:54.396238    2581 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45_kserve-ci-e2e-test_9836ada4-8a3e-40fd-b74f-d6270d1f1328_0 in pod sandbox 5e6b77681f43d64e96b26bc21042da90ff1c406030d1608dd7a4cb3993f6f4a8 from index: no such id: '5368ec11f5fc5901b082f0c995cefdaf2ef8b7ab49e2cd7edb224982e448334f'" containerID="5368ec11f5fc5901b082f0c995cefdaf2ef8b7ab49e2cd7edb224982e448334f"
Jul 28 18:40:54.396331 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:54.396277    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5368ec11f5fc5901b082f0c995cefdaf2ef8b7ab49e2cd7edb224982e448334f"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45_kserve-ci-e2e-test_9836ada4-8a3e-40fd-b74f-d6270d1f1328_0 in pod sandbox 5e6b77681f43d64e96b26bc21042da90ff1c406030d1608dd7a4cb3993f6f4a8 from index: no such id: '5368ec11f5fc5901b082f0c995cefdaf2ef8b7ab49e2cd7edb224982e448334f'"
Jul 28 18:40:55.390575 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:55.390542    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45_9836ada4-8a3e-40fd-b74f-d6270d1f1328/storage-initializer/1.log"
Jul 28 18:40:56.955698 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:56.955659    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45"]
Jul 28 18:40:56.965075 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:56.965041    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:40:57.085849 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:57.085824    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45_9836ada4-8a3e-40fd-b74f-d6270d1f1328/storage-initializer/1.log"
Jul 28 18:40:57.089405 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:57.089388    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45"
Jul 28 18:40:57.193258 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:57.193220    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9836ada4-8a3e-40fd-b74f-d6270d1f1328-kube-rbac-proxy-sar-config\") pod \"9836ada4-8a3e-40fd-b74f-d6270d1f1328\" (UID: \"9836ada4-8a3e-40fd-b74f-d6270d1f1328\") "
Jul 28 18:40:57.193258 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:57.193268    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9836ada4-8a3e-40fd-b74f-d6270d1f1328-proxy-tls\") pod \"9836ada4-8a3e-40fd-b74f-d6270d1f1328\" (UID: \"9836ada4-8a3e-40fd-b74f-d6270d1f1328\") "
Jul 28 18:40:57.193599 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:57.193337    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmgw7\" (UniqueName: \"kubernetes.io/projected/9836ada4-8a3e-40fd-b74f-d6270d1f1328-kube-api-access-mmgw7\") pod \"9836ada4-8a3e-40fd-b74f-d6270d1f1328\" (UID: \"9836ada4-8a3e-40fd-b74f-d6270d1f1328\") "
Jul 28 18:40:57.193599 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:57.193359    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9836ada4-8a3e-40fd-b74f-d6270d1f1328-kserve-provision-location\") pod \"9836ada4-8a3e-40fd-b74f-d6270d1f1328\" (UID: \"9836ada4-8a3e-40fd-b74f-d6270d1f1328\") "
Jul 28 18:40:57.193717 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:57.193632    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9836ada4-8a3e-40fd-b74f-d6270d1f1328-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9836ada4-8a3e-40fd-b74f-d6270d1f1328" (UID: "9836ada4-8a3e-40fd-b74f-d6270d1f1328"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:40:57.193717 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:57.193678    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9836ada4-8a3e-40fd-b74f-d6270d1f1328-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9836ada4-8a3e-40fd-b74f-d6270d1f1328" (UID: "9836ada4-8a3e-40fd-b74f-d6270d1f1328"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:40:57.195509 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:57.195484    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9836ada4-8a3e-40fd-b74f-d6270d1f1328-kube-api-access-mmgw7" (OuterVolumeSpecName: "kube-api-access-mmgw7") pod "9836ada4-8a3e-40fd-b74f-d6270d1f1328" (UID: "9836ada4-8a3e-40fd-b74f-d6270d1f1328"). InnerVolumeSpecName "kube-api-access-mmgw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:40:57.195612 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:57.195523    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9836ada4-8a3e-40fd-b74f-d6270d1f1328-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9836ada4-8a3e-40fd-b74f-d6270d1f1328" (UID: "9836ada4-8a3e-40fd-b74f-d6270d1f1328"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:40:57.294019 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:57.293921    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9836ada4-8a3e-40fd-b74f-d6270d1f1328-kserve-provision-location\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:40:57.294019 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:57.293965    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9836ada4-8a3e-40fd-b74f-d6270d1f1328-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:40:57.294019 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:57.293975    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9836ada4-8a3e-40fd-b74f-d6270d1f1328-proxy-tls\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:40:57.294019 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:57.293984    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmgw7\" (UniqueName: \"kubernetes.io/projected/9836ada4-8a3e-40fd-b74f-d6270d1f1328-kube-api-access-mmgw7\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:40:57.398975 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:57.398930    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45_9836ada4-8a3e-40fd-b74f-d6270d1f1328/storage-initializer/1.log"
Jul 28 18:40:57.399136 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:57.399059    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45"
Jul 28 18:40:57.399136 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:57.399075    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45" event={"ID":"9836ada4-8a3e-40fd-b74f-d6270d1f1328","Type":"ContainerDied","Data":"5e6b77681f43d64e96b26bc21042da90ff1c406030d1608dd7a4cb3993f6f4a8"}
Jul 28 18:40:57.399215 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:57.399145    2581 scope.go:117] "RemoveContainer" containerID="7792ef646698a9121df3fe38bdabca03dc81b6c1172a74a582a92a8a57297d03"
Jul 28 18:40:57.435666 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:57.435634    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45"]
Jul 28 18:40:57.441478 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:57.441448    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-689d488888-c4j45"]
Jul 28 18:40:58.290300 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:58.290261    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9836ada4-8a3e-40fd-b74f-d6270d1f1328" path="/var/lib/kubelet/pods/9836ada4-8a3e-40fd-b74f-d6270d1f1328/volumes"
Jul 28 18:40:59.401240 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.401204    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hhc22/must-gather-gpdnp"]
Jul 28 18:40:59.401642 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.401441    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ac4efe6-3f27-405d-9da4-acf6ac07601f" containerName="storage-initializer"
Jul 28 18:40:59.401642 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.401451    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ac4efe6-3f27-405d-9da4-acf6ac07601f" containerName="storage-initializer"
Jul 28 18:40:59.401642 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.401461    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9836ada4-8a3e-40fd-b74f-d6270d1f1328" containerName="storage-initializer"
Jul 28 18:40:59.401642 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.401467    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9836ada4-8a3e-40fd-b74f-d6270d1f1328" containerName="storage-initializer"
Jul 28 18:40:59.401642 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.401474    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9836ada4-8a3e-40fd-b74f-d6270d1f1328" containerName="storage-initializer"
Jul 28 18:40:59.401642 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.401480    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9836ada4-8a3e-40fd-b74f-d6270d1f1328" containerName="storage-initializer"
Jul 28 18:40:59.401642 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.401494    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ac4efe6-3f27-405d-9da4-acf6ac07601f" containerName="kube-rbac-proxy"
Jul 28 18:40:59.401642 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.401499    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ac4efe6-3f27-405d-9da4-acf6ac07601f" containerName="kube-rbac-proxy"
Jul 28 18:40:59.401642 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.401505    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ac4efe6-3f27-405d-9da4-acf6ac07601f" containerName="kserve-container"
Jul 28 18:40:59.401642 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.401510    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ac4efe6-3f27-405d-9da4-acf6ac07601f" containerName="kserve-container"
Jul 28 18:40:59.401642 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.401553    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ac4efe6-3f27-405d-9da4-acf6ac07601f" containerName="kube-rbac-proxy"
Jul 28 18:40:59.401642 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.401560    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ac4efe6-3f27-405d-9da4-acf6ac07601f" containerName="kserve-container"
Jul 28 18:40:59.401642 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.401565    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9836ada4-8a3e-40fd-b74f-d6270d1f1328" containerName="storage-initializer"
Jul 28 18:40:59.401642 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.401635    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9836ada4-8a3e-40fd-b74f-d6270d1f1328" containerName="storage-initializer"
Jul 28 18:40:59.405615 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.405594    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hhc22/must-gather-gpdnp"
Jul 28 18:40:59.408196 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.408172    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hhc22\"/\"kube-root-ca.crt\""
Jul 28 18:40:59.408311 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.408177    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hhc22\"/\"openshift-service-ca.crt\""
Jul 28 18:40:59.413341 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.413317    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hhc22/must-gather-gpdnp"]
Jul 28 18:40:59.426330 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.426305    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:40:59.490910 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.490872    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:40:59.508443 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.508408    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5552f162-797a-4554-978c-f9bc2161de70-must-gather-output\") pod \"must-gather-gpdnp\" (UID: \"5552f162-797a-4554-978c-f9bc2161de70\") " pod="openshift-must-gather-hhc22/must-gather-gpdnp"
Jul 28 18:40:59.508591 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.508462    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f9dn\" (UniqueName: \"kubernetes.io/projected/5552f162-797a-4554-978c-f9bc2161de70-kube-api-access-2f9dn\") pod \"must-gather-gpdnp\" (UID: \"5552f162-797a-4554-978c-f9bc2161de70\") " pod="openshift-must-gather-hhc22/must-gather-gpdnp"
Jul 28 18:40:59.609048 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.608993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5552f162-797a-4554-978c-f9bc2161de70-must-gather-output\") pod \"must-gather-gpdnp\" (UID: \"5552f162-797a-4554-978c-f9bc2161de70\") " pod="openshift-must-gather-hhc22/must-gather-gpdnp"
Jul 28 18:40:59.609226 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.609084    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f9dn\" (UniqueName: \"kubernetes.io/projected/5552f162-797a-4554-978c-f9bc2161de70-kube-api-access-2f9dn\") pod \"must-gather-gpdnp\" (UID: \"5552f162-797a-4554-978c-f9bc2161de70\") " pod="openshift-must-gather-hhc22/must-gather-gpdnp"
Jul 28 18:40:59.609416 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.609395    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5552f162-797a-4554-978c-f9bc2161de70-must-gather-output\") pod \"must-gather-gpdnp\" (UID: \"5552f162-797a-4554-978c-f9bc2161de70\") " pod="openshift-must-gather-hhc22/must-gather-gpdnp"
Jul 28 18:40:59.617436 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.617413    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f9dn\" (UniqueName: \"kubernetes.io/projected/5552f162-797a-4554-978c-f9bc2161de70-kube-api-access-2f9dn\") pod \"must-gather-gpdnp\" (UID: \"5552f162-797a-4554-978c-f9bc2161de70\") " pod="openshift-must-gather-hhc22/must-gather-gpdnp"
Jul 28 18:40:59.714650 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.714545    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hhc22/must-gather-gpdnp"
Jul 28 18:40:59.846826 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.846755    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hhc22/must-gather-gpdnp"]
Jul 28 18:40:59.850809 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:40:59.850773    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5552f162_797a_4554_978c_f9bc2161de70.slice/crio-235f919609f07b565625a6906b9a495684e113bb39c26291dd3c1119076a9b62 WatchSource:0}: Error finding container 235f919609f07b565625a6906b9a495684e113bb39c26291dd3c1119076a9b62: Status 404 returned error can't find the container with id 235f919609f07b565625a6906b9a495684e113bb39c26291dd3c1119076a9b62
Jul 28 18:40:59.852452 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:40:59.852433    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:41:00.412768 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:00.412734    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hhc22/must-gather-gpdnp" event={"ID":"5552f162-797a-4554-978c-f9bc2161de70","Type":"ContainerStarted","Data":"235f919609f07b565625a6906b9a495684e113bb39c26291dd3c1119076a9b62"}
Jul 28 18:41:06.439123 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:06.439083    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hhc22/must-gather-gpdnp" event={"ID":"5552f162-797a-4554-978c-f9bc2161de70","Type":"ContainerStarted","Data":"f9be6d261164925ebec6a27f387891ed8c47ae1f70f6f33d0dda9beb24c38c41"}
Jul 28 18:41:06.439123 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:06.439132    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hhc22/must-gather-gpdnp" event={"ID":"5552f162-797a-4554-978c-f9bc2161de70","Type":"ContainerStarted","Data":"d68cb08c304049802b1e154a1ebf9f0e0c9e549075464c6c150c7d71f8ad4004"}
Jul 28 18:41:06.456191 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:06.456127    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hhc22/must-gather-gpdnp" podStartSLOduration=1.428885801 podStartE2EDuration="7.456106329s" totalImagesPullingTime="6.027220528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:40:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:41:06.455600624 +0000 UTC m=+5514.730217057" watchObservedRunningTime="2026-07-28 18:41:06.456106329 +0000 UTC m=+5514.730722762"
Jul 28 18:41:32.539833 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:32.539796    2581 generic.go:356] "Generic (PLEG): container finished" podID="5552f162-797a-4554-978c-f9bc2161de70" containerID="d68cb08c304049802b1e154a1ebf9f0e0c9e549075464c6c150c7d71f8ad4004" exitCode=0
Jul 28 18:41:32.540299 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:32.539872    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hhc22/must-gather-gpdnp" event={"ID":"5552f162-797a-4554-978c-f9bc2161de70","Type":"ContainerDied","Data":"d68cb08c304049802b1e154a1ebf9f0e0c9e549075464c6c150c7d71f8ad4004"}
Jul 28 18:41:32.540299 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:32.540206    2581 scope.go:117] "RemoveContainer" containerID="d68cb08c304049802b1e154a1ebf9f0e0c9e549075464c6c150c7d71f8ad4004"
Jul 28 18:41:33.115902 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:33.115873    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-hhc22_must-gather-gpdnp_5552f162-797a-4554-978c-f9bc2161de70/gather/0.log"
Jul 28 18:41:33.640856 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:33.640819    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:41:33.795209 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:33.795177    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:41:33.848133 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:33.848100    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:41:36.501230 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:36.501199    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jwmpm_189dbf08-647d-42a8-975f-39838ad4edf9/global-pull-secret-syncer/0.log"
Jul 28 18:41:36.558545 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:36.558513    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log"
Jul 28 18:41:36.736020 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:36.735987    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-231.ec2.internal_83f9b4a12578fa1633b19325b5319dc4/haproxy/0.log"
Jul 28 18:41:38.655227 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:38.655186    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-hhc22/must-gather-gpdnp"]
Jul 28 18:41:38.655706 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:38.655390    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-hhc22/must-gather-gpdnp" podUID="5552f162-797a-4554-978c-f9bc2161de70" containerName="copy" containerID="cri-o://f9be6d261164925ebec6a27f387891ed8c47ae1f70f6f33d0dda9beb24c38c41" gracePeriod=2
Jul 28 18:41:38.657384 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:38.657363    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-hhc22/must-gather-gpdnp"]
Jul 28 18:41:38.657739 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:41:38.657708    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-gpdnp\" is forbidden: User \"system:node:ip-10-0-138-231.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-hhc22\": no relationship found between node 'ip-10-0-138-231.ec2.internal' and this object" podUID="5552f162-797a-4554-978c-f9bc2161de70" pod="openshift-must-gather-hhc22/must-gather-gpdnp"
Jul 28 18:41:38.886216 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:38.886193    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-hhc22_must-gather-gpdnp_5552f162-797a-4554-978c-f9bc2161de70/copy/0.log"
Jul 28 18:41:38.888841 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:38.888823    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hhc22/must-gather-gpdnp"
Jul 28 18:41:38.891172 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:41:38.891148    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-gpdnp\" is forbidden: User \"system:node:ip-10-0-138-231.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-hhc22\": no relationship found between node 'ip-10-0-138-231.ec2.internal' and this object" podUID="5552f162-797a-4554-978c-f9bc2161de70" pod="openshift-must-gather-hhc22/must-gather-gpdnp"
Jul 28 18:41:39.035692 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:39.035585    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5552f162-797a-4554-978c-f9bc2161de70-must-gather-output\") pod \"5552f162-797a-4554-978c-f9bc2161de70\" (UID: \"5552f162-797a-4554-978c-f9bc2161de70\") "
Jul 28 18:41:39.035692 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:39.035691    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2f9dn\" (UniqueName: \"kubernetes.io/projected/5552f162-797a-4554-978c-f9bc2161de70-kube-api-access-2f9dn\") pod \"5552f162-797a-4554-978c-f9bc2161de70\" (UID: \"5552f162-797a-4554-978c-f9bc2161de70\") "
Jul 28 18:41:39.037168 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:39.037144    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5552f162-797a-4554-978c-f9bc2161de70-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "5552f162-797a-4554-978c-f9bc2161de70" (UID: "5552f162-797a-4554-978c-f9bc2161de70"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:41:39.037862 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:39.037828    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5552f162-797a-4554-978c-f9bc2161de70-kube-api-access-2f9dn" (OuterVolumeSpecName: "kube-api-access-2f9dn") pod "5552f162-797a-4554-978c-f9bc2161de70" (UID: "5552f162-797a-4554-978c-f9bc2161de70"). InnerVolumeSpecName "kube-api-access-2f9dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:41:39.136883 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:39.136833    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f9dn\" (UniqueName: \"kubernetes.io/projected/5552f162-797a-4554-978c-f9bc2161de70-kube-api-access-2f9dn\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:41:39.136883 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:39.136877    2581 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5552f162-797a-4554-978c-f9bc2161de70-must-gather-output\") on node \"ip-10-0-138-231.ec2.internal\" DevicePath \"\""
Jul 28 18:41:39.432346 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:39.432299    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:41:39.491929 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:39.491884    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9pw_3b4343ba-84d3-4d95-8a25-e97b6cf88830/konnectivity-agent/0.log" limit=1024
Jul 28 18:41:39.564217 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:39.564194    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-hhc22_must-gather-gpdnp_5552f162-797a-4554-978c-f9bc2161de70/copy/0.log"
Jul 28 18:41:39.564546 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:39.564524    2581 generic.go:356] "Generic (PLEG): container finished" podID="5552f162-797a-4554-978c-f9bc2161de70" containerID="f9be6d261164925ebec6a27f387891ed8c47ae1f70f6f33d0dda9beb24c38c41" exitCode=143
Jul 28 18:41:39.564617 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:39.564563    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hhc22/must-gather-gpdnp"
Jul 28 18:41:39.564617 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:39.564576    2581 scope.go:117] "RemoveContainer" containerID="f9be6d261164925ebec6a27f387891ed8c47ae1f70f6f33d0dda9beb24c38c41"
Jul 28 18:41:39.567374 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:41:39.567343    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-gpdnp\" is forbidden: User \"system:node:ip-10-0-138-231.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-hhc22\": no relationship found between node 'ip-10-0-138-231.ec2.internal' and this object" podUID="5552f162-797a-4554-978c-f9bc2161de70" pod="openshift-must-gather-hhc22/must-gather-gpdnp"
Jul 28 18:41:39.571741 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:39.571719    2581 scope.go:117] "RemoveContainer" containerID="d68cb08c304049802b1e154a1ebf9f0e0c9e549075464c6c150c7d71f8ad4004"
Jul 28 18:41:39.575478 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:41:39.575450    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-gpdnp\" is forbidden: User \"system:node:ip-10-0-138-231.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-hhc22\": no relationship found between node 'ip-10-0-138-231.ec2.internal' and this object" podUID="5552f162-797a-4554-978c-f9bc2161de70" pod="openshift-must-gather-hhc22/must-gather-gpdnp"
Jul 28 18:41:39.583504 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:39.583483    2581 scope.go:117] "RemoveContainer" containerID="f9be6d261164925ebec6a27f387891ed8c47ae1f70f6f33d0dda9beb24c38c41"
Jul 28 18:41:39.583829 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:41:39.583809    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9be6d261164925ebec6a27f387891ed8c47ae1f70f6f33d0dda9beb24c38c41\": container with ID starting with f9be6d261164925ebec6a27f387891ed8c47ae1f70f6f33d0dda9beb24c38c41 not found: ID does not exist" containerID="f9be6d261164925ebec6a27f387891ed8c47ae1f70f6f33d0dda9beb24c38c41"
Jul 28 18:41:39.583880 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:39.583838    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9be6d261164925ebec6a27f387891ed8c47ae1f70f6f33d0dda9beb24c38c41"} err="failed to get container status \"f9be6d261164925ebec6a27f387891ed8c47ae1f70f6f33d0dda9beb24c38c41\": rpc error: code = NotFound desc = could not find container \"f9be6d261164925ebec6a27f387891ed8c47ae1f70f6f33d0dda9beb24c38c41\": container with ID starting with f9be6d261164925ebec6a27f387891ed8c47ae1f70f6f33d0dda9beb24c38c41 not found: ID does not exist"
Jul 28 18:41:39.583880 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:39.583859    2581 scope.go:117] "RemoveContainer" containerID="d68cb08c304049802b1e154a1ebf9f0e0c9e549075464c6c150c7d71f8ad4004"
Jul 28 18:41:39.584154 ip-10-0-138-231 kubenswrapper[2581]: E0728 18:41:39.584136    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d68cb08c304049802b1e154a1ebf9f0e0c9e549075464c6c150c7d71f8ad4004\": container with ID starting with d68cb08c304049802b1e154a1ebf9f0e0c9e549075464c6c150c7d71f8ad4004 not found: ID does not exist" containerID="d68cb08c304049802b1e154a1ebf9f0e0c9e549075464c6c150c7d71f8ad4004"
Jul 28 18:41:39.584208 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:39.584158    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d68cb08c304049802b1e154a1ebf9f0e0c9e549075464c6c150c7d71f8ad4004"} err="failed to get container status \"d68cb08c304049802b1e154a1ebf9f0e0c9e549075464c6c150c7d71f8ad4004\": rpc error: code = NotFound desc = could not find container \"d68cb08c304049802b1e154a1ebf9f0e0c9e549075464c6c150c7d71f8ad4004\": container with ID starting with d68cb08c304049802b1e154a1ebf9f0e0c9e549075464c6c150c7d71f8ad4004 not found: ID does not exist"
Jul 28 18:41:40.294791 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:40.294707    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5552f162-797a-4554-978c-f9bc2161de70" path="/var/lib/kubelet/pods/5552f162-797a-4554-978c-f9bc2161de70/volumes"
Jul 28 18:41:41.272867 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:41.272838    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qw4bp_34af819b-df81-4984-81ca-d05c55376b97/node-exporter/0.log"
Jul 28 18:41:41.296999 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:41.296974    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qw4bp_34af819b-df81-4984-81ca-d05c55376b97/kube-rbac-proxy/0.log"
Jul 28 18:41:41.323711 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:41.323680    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qw4bp_34af819b-df81-4984-81ca-d05c55376b97/init-textfile/0.log"
Jul 28 18:41:44.750014 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:44.749979    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fsvbn/perf-node-gather-daemonset-fz7gd"]
Jul 28 18:41:44.750435 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:44.750222    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5552f162-797a-4554-978c-f9bc2161de70" containerName="gather"
Jul 28 18:41:44.750435 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:44.750232    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5552f162-797a-4554-978c-f9bc2161de70" containerName="gather"
Jul 28 18:41:44.750435 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:44.750241    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5552f162-797a-4554-978c-f9bc2161de70" containerName="copy"
Jul 28 18:41:44.750435 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:44.750248    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5552f162-797a-4554-978c-f9bc2161de70" containerName="copy"
Jul 28 18:41:44.750435 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:44.750296    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5552f162-797a-4554-978c-f9bc2161de70" containerName="copy"
Jul 28 18:41:44.750435 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:44.750305    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5552f162-797a-4554-978c-f9bc2161de70" containerName="gather"
Jul 28 18:41:44.753051 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:44.753034    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-fz7gd"
Jul 28 18:41:44.766365 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:44.766342    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fsvbn\"/\"openshift-service-ca.crt\""
Jul 28 18:41:44.766498 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:44.766387    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fsvbn\"/\"kube-root-ca.crt\""
Jul 28 18:41:44.767142 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:44.767126    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fsvbn\"/\"default-dockercfg-h49r6\""
Jul 28 18:41:44.782488 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:44.782463    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fsvbn/perf-node-gather-daemonset-fz7gd"]
Jul 28 18:41:44.879748 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:44.879702    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/66703924-3f2c-4838-9e78-da5f49e5c7ab-proc\") pod \"perf-node-gather-daemonset-fz7gd\" (UID: \"66703924-3f2c-4838-9e78-da5f49e5c7ab\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-fz7gd"
Jul 28 18:41:44.879929 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:44.879756    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/66703924-3f2c-4838-9e78-da5f49e5c7ab-podres\") pod \"perf-node-gather-daemonset-fz7gd\" (UID: \"66703924-3f2c-4838-9e78-da5f49e5c7ab\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-fz7gd"
Jul 28 18:41:44.879929 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:44.879842    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfm4m\" (UniqueName: \"kubernetes.io/projected/66703924-3f2c-4838-9e78-da5f49e5c7ab-kube-api-access-vfm4m\") pod \"perf-node-gather-daemonset-fz7gd\" (UID: \"66703924-3f2c-4838-9e78-da5f49e5c7ab\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-fz7gd"
Jul 28 18:41:44.879929 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:44.879877    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66703924-3f2c-4838-9e78-da5f49e5c7ab-lib-modules\") pod \"perf-node-gather-daemonset-fz7gd\" (UID: \"66703924-3f2c-4838-9e78-da5f49e5c7ab\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-fz7gd"
Jul 28 18:41:44.879929 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:44.879897    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66703924-3f2c-4838-9e78-da5f49e5c7ab-sys\") pod \"perf-node-gather-daemonset-fz7gd\" (UID: \"66703924-3f2c-4838-9e78-da5f49e5c7ab\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-fz7gd"
Jul 28 18:41:44.980609 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:44.980563    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66703924-3f2c-4838-9e78-da5f49e5c7ab-lib-modules\") pod \"perf-node-gather-daemonset-fz7gd\" (UID: \"66703924-3f2c-4838-9e78-da5f49e5c7ab\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-fz7gd"
Jul 28 18:41:44.980609 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:44.980611    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66703924-3f2c-4838-9e78-da5f49e5c7ab-sys\") pod \"perf-node-gather-daemonset-fz7gd\" (UID: \"66703924-3f2c-4838-9e78-da5f49e5c7ab\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-fz7gd"
Jul 28 18:41:44.980880 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:44.980648    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/66703924-3f2c-4838-9e78-da5f49e5c7ab-proc\") pod \"perf-node-gather-daemonset-fz7gd\" (UID: \"66703924-3f2c-4838-9e78-da5f49e5c7ab\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-fz7gd"
Jul 28 18:41:44.980880 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:44.980687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/66703924-3f2c-4838-9e78-da5f49e5c7ab-podres\") pod \"perf-node-gather-daemonset-fz7gd\" (UID: \"66703924-3f2c-4838-9e78-da5f49e5c7ab\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-fz7gd"
Jul 28 18:41:44.980880 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:44.980758    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66703924-3f2c-4838-9e78-da5f49e5c7ab-lib-modules\") pod \"perf-node-gather-daemonset-fz7gd\" (UID: \"66703924-3f2c-4838-9e78-da5f49e5c7ab\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-fz7gd"
Jul 28 18:41:44.980880 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:44.980773    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/66703924-3f2c-4838-9e78-da5f49e5c7ab-proc\") pod \"perf-node-gather-daemonset-fz7gd\" (UID: \"66703924-3f2c-4838-9e78-da5f49e5c7ab\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-fz7gd"
Jul 28 18:41:44.980880 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:44.980758    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66703924-3f2c-4838-9e78-da5f49e5c7ab-sys\") pod \"perf-node-gather-daemonset-fz7gd\" (UID: \"66703924-3f2c-4838-9e78-da5f49e5c7ab\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-fz7gd"
Jul 28 18:41:44.980880 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:44.980810    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/66703924-3f2c-4838-9e78-da5f49e5c7ab-podres\") pod \"perf-node-gather-daemonset-fz7gd\" (UID: \"66703924-3f2c-4838-9e78-da5f49e5c7ab\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-fz7gd"
Jul 28 18:41:44.980880 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:44.980819    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfm4m\" (UniqueName: \"kubernetes.io/projected/66703924-3f2c-4838-9e78-da5f49e5c7ab-kube-api-access-vfm4m\") pod \"perf-node-gather-daemonset-fz7gd\" (UID: \"66703924-3f2c-4838-9e78-da5f49e5c7ab\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-fz7gd"
Jul 28 18:41:44.990543 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:44.990523    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfm4m\" (UniqueName: \"kubernetes.io/projected/66703924-3f2c-4838-9e78-da5f49e5c7ab-kube-api-access-vfm4m\") pod \"perf-node-gather-daemonset-fz7gd\" (UID: \"66703924-3f2c-4838-9e78-da5f49e5c7ab\") " pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-fz7gd"
Jul 28 18:41:45.062620 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:45.062511    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-fz7gd"
Jul 28 18:41:45.188327 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:45.188292    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fsvbn/perf-node-gather-daemonset-fz7gd"]
Jul 28 18:41:45.191558 ip-10-0-138-231 kubenswrapper[2581]: W0728 18:41:45.191527    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod66703924_3f2c_4838_9e78_da5f49e5c7ab.slice/crio-ed9f1775205f2bfe7e0ab260652c64f0242e780502294be960f2f25dbc4d5727 WatchSource:0}: Error finding container ed9f1775205f2bfe7e0ab260652c64f0242e780502294be960f2f25dbc4d5727: Status 404 returned error can't find the container with id ed9f1775205f2bfe7e0ab260652c64f0242e780502294be960f2f25dbc4d5727
Jul 28 18:41:45.587970 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:45.587919    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-fz7gd" event={"ID":"66703924-3f2c-4838-9e78-da5f49e5c7ab","Type":"ContainerStarted","Data":"3854661a8e43e1b9de2acc4ed0ce13adcca10d84c8bcaa7f936f3944daedba57"}
Jul 28 18:41:45.588158 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:45.587987    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-fz7gd" event={"ID":"66703924-3f2c-4838-9e78-da5f49e5c7ab","Type":"ContainerStarted","Data":"ed9f1775205f2bfe7e0ab260652c64f0242e780502294be960f2f25dbc4d5727"}
Jul 28 18:41:45.588158 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:45.588018    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-fz7gd"
Jul 28 18:41:45.604202 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:45.604150    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-fz7gd" podStartSLOduration=1.604133765 podStartE2EDuration="1.604133765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:41:45.603123925 +0000 UTC m=+5553.877740354" watchObservedRunningTime="2026-07-28 18:41:45.604133765 +0000 UTC m=+5553.878750219"
Jul 28 18:41:45.916846 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:45.916817    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mvdj9_81dc569e-3e52-4bb3-8f79-d58b183bb100/dns/0.log"
Jul 28 18:41:45.936482 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:45.936451    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mvdj9_81dc569e-3e52-4bb3-8f79-d58b183bb100/kube-rbac-proxy/0.log"
Jul 28 18:41:45.965115 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:45.965082    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dbnhf_601b853c-8800-428b-89f6-1cb7b378600c/dns-node-resolver/0.log"
Jul 28 18:41:46.516582 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:46.516551    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ffvhq_6d4cfcb5-839a-431f-946c-6b5ac4e6c61e/node-ca/0.log"
Jul 28 18:41:47.248651 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:47.248614    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5555ff8d6c-6mzsx_496b869a-13ac-4171-8068-7c839c75b1e9/router/0.log"
Jul 28 18:41:47.621293 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:47.621193    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-h6pjb_baf5cf61-87a1-442f-981a-8d24b1197ec8/serve-healthcheck-canary/0.log"
Jul 28 18:41:48.044321 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:48.044294    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z65fp_137885a5-ec67-4cfd-baa1-2a43a9389bb1/kube-rbac-proxy/0.log"
Jul 28 18:41:48.065274 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:48.065241    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z65fp_137885a5-ec67-4cfd-baa1-2a43a9389bb1/exporter/0.log"
Jul 28 18:41:48.087670 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:48.087634    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z65fp_137885a5-ec67-4cfd-baa1-2a43a9389bb1/extractor/0.log"
Jul 28 18:41:50.596283 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:50.596250    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-86b8n_a1da0f27-245f-4a42-816c-30ddc49048f5/server/0.log"
Jul 28 18:41:51.303485 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:51.303450    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-f5q4d_e6604f28-7469-40d5-a52b-ddc3fb14156f/seaweedfs/0.log"
Jul 28 18:41:51.357508 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:51.357475    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-pldmn_9e9cf802-1539-4f3a-a439-6894d2ccff0c/seaweedfs-tls-serving/0.log"
Jul 28 18:41:51.602313 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:51.602225    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fsvbn/perf-node-gather-daemonset-fz7gd"
Jul 28 18:41:56.920050 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:56.920021    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-njk8v_1e459f08-3f93-48bd-bb45-b950f416c87f/kube-multus-additional-cni-plugins/0.log"
Jul 28 18:41:56.943381 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:56.943346    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-njk8v_1e459f08-3f93-48bd-bb45-b950f416c87f/egress-router-binary-copy/0.log"
Jul 28 18:41:56.967379 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:56.967347    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-njk8v_1e459f08-3f93-48bd-bb45-b950f416c87f/cni-plugins/0.log"
Jul 28 18:41:56.990217 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:56.990190    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-njk8v_1e459f08-3f93-48bd-bb45-b950f416c87f/bond-cni-plugin/0.log"
Jul 28 18:41:57.013141 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:57.013110    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-njk8v_1e459f08-3f93-48bd-bb45-b950f416c87f/routeoverride-cni/0.log"
Jul 28 18:41:57.042061 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:57.042036    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-njk8v_1e459f08-3f93-48bd-bb45-b950f416c87f/whereabouts-cni-bincopy/0.log"
Jul 28 18:41:57.066626 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:57.066600    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-njk8v_1e459f08-3f93-48bd-bb45-b950f416c87f/whereabouts-cni/0.log"
Jul 28 18:41:57.328627 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:57.328530    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r5x5j_5a4cb725-44da-4042-8bc6-8f09251a6325/kube-multus/0.log"
Jul 28 18:41:57.447774 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:57.447738    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zkf8x_9a7b47e4-993e-4527-a00f-4ebbbc050348/network-metrics-daemon/0.log"
Jul 28 18:41:57.469923 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:57.469897    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zkf8x_9a7b47e4-993e-4527-a00f-4ebbbc050348/kube-rbac-proxy/0.log"
Jul 28 18:41:58.221961 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:58.221921    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-controller/0.log"
Jul 28 18:41:58.242875 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:58.242846    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/1.log"
Jul 28 18:41:58.243066 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:58.242848    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovn-acl-logging/0.log"
Jul 28 18:41:58.263855 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:58.263830    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/kube-rbac-proxy-node/0.log"
Jul 28 18:41:58.289345 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:58.289290    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 18:41:58.314386 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:58.314357    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/northd/0.log"
Jul 28 18:41:58.339455 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:58.339424    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/nbdb/0.log"
Jul 28 18:41:58.361350 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:58.361322    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/sbdb/0.log"
Jul 28 18:41:58.410639 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:58.410589    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9l7dh_b0a6ecd5-1b96-4003-ba79-fe147a67c757/ovnkube-controller/0.log"
Jul 28 18:41:59.830266 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:41:59.830235    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-2nt29_33068506-3820-4d64-bc6b-33ea180dca70/network-check-target-container/0.log"
Jul 28 18:42:00.786271 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:42:00.786240    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-pp6xv_8188b00b-a96c-4bd3-a9b9-35fc129184d5/iptables-alerter/0.log"
Jul 28 18:42:01.390964 ip-10-0-138-231 kubenswrapper[2581]: I0728 18:42:01.390917    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-jhs8c_ed21aa75-f994-4548-93f7-99c172998ea1/tuned/0.log"