Jul 31 14:46:30.104890 ip-10-0-142-219 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 31 14:46:30.104903 ip-10-0-142-219 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 31 14:46:30.104913 ip-10-0-142-219 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 31 14:46:30.105232 ip-10-0-142-219 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 31 14:46:41.442298 ip-10-0-142-219 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 31 14:46:41.442340 ip-10-0-142-219 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 9074b0ff02cd48908b5d60d43d163f91 --
Jul 31 14:48:23.573052 ip-10-0-142-219 systemd[1]: Starting Kubernetes Kubelet...
Jul 31 14:48:23.976498 ip-10-0-142-219 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 14:48:23.976498 ip-10-0-142-219 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 31 14:48:23.976498 ip-10-0-142-219 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 14:48:23.976498 ip-10-0-142-219 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 31 14:48:23.976498 ip-10-0-142-219 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 14:48:23.977217 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.976569    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 31 14:48:23.980433 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980405    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:48:23.980433 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980434    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:48:23.980433 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980438    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:48:23.980433 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980442    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:48:23.980433 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980445    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:48:23.980583 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980448    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:48:23.980583 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980451    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:48:23.980583 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980454    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:48:23.980583 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980457    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:48:23.980583 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980462    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:48:23.980583 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980466    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:48:23.980583 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980470    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:48:23.980583 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980472    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:48:23.980583 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980475    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:48:23.980583 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980478    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:48:23.980583 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980480    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:48:23.980583 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980490    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:48:23.980583 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980492    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:48:23.980583 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980495    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:48:23.980583 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980498    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:48:23.980583 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980500    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:48:23.980583 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980503    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:48:23.980583 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980505    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:48:23.980583 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980507    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:48:23.980583 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980510    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:48:23.981076 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980512    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:48:23.981076 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980515    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:48:23.981076 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980518    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:48:23.981076 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980521    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:48:23.981076 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980523    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:48:23.981076 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980526    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:48:23.981076 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980529    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:48:23.981076 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980531    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:48:23.981076 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980534    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:48:23.981076 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980537    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:48:23.981076 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980539    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:48:23.981076 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980541    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:48:23.981076 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980545    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:48:23.981076 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980548    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:48:23.981076 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980550    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:48:23.981076 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980553    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:48:23.981076 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980555    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:48:23.981076 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980558    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:48:23.981076 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980560    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:48:23.981076 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980563    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:48:23.981553 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980566    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:48:23.981553 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980570    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:48:23.981553 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980573    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:48:23.981553 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980576    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:48:23.981553 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980578    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:48:23.981553 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980581    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:48:23.981553 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980583    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:48:23.981553 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980588    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:48:23.981553 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980592    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:48:23.981553 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980595    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:48:23.981553 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980598    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:48:23.981553 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980601    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:48:23.981553 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980604    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:48:23.981553 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980606    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:48:23.981553 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980609    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:48:23.981553 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980612    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:48:23.981553 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980614    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:48:23.981553 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980617    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:48:23.981553 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980619    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:48:23.981553 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980622    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:48:23.982048 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980624    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:48:23.982048 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980627    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:48:23.982048 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980629    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:48:23.982048 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980631    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:48:23.982048 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980634    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:48:23.982048 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980638    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:48:23.982048 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980641    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:48:23.982048 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980643    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:48:23.982048 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980646    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:48:23.982048 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980648    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:48:23.982048 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980651    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:48:23.982048 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980653    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:48:23.982048 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980656    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:48:23.982048 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980658    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:48:23.982048 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980661    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:48:23.982048 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980665    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:48:23.982048 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980669    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:48:23.982048 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980671    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:48:23.982048 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980674    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:48:23.982489 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980677    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:48:23.982489 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980679    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:48:23.982489 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980683    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:48:23.982489 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980686    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:48:23.982489 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980688    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:48:23.982489 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980691    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:48:23.982489 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980693    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:48:23.982489 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980696    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:48:23.982489 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980698    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:48:23.982489 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980701    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:48:23.982489 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980703    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:48:23.982489 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980705    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:48:23.982489 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980708    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:48:23.982489 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980711    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:48:23.982489 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.980713    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:48:23.982489 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981411    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:48:23.982489 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981418    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:48:23.982489 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981421    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:48:23.982489 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981424    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:48:23.982489 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981426    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:48:23.982990 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981429    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:48:23.982990 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981431    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:48:23.982990 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981434    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:48:23.982990 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981436    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:48:23.982990 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981439    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:48:23.982990 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981441    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:48:23.982990 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981444    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:48:23.982990 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981446    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:48:23.982990 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981452    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:48:23.982990 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981455    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:48:23.982990 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981458    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:48:23.982990 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981462    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:48:23.982990 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981466    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:48:23.982990 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981469    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:48:23.982990 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981472    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:48:23.982990 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981475    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:48:23.982990 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981477    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:48:23.982990 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981480    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:48:23.982990 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981483    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:48:23.982990 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981486    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:48:23.983476 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981488    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:48:23.983476 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981493    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:48:23.983476 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981496    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:48:23.983476 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981498    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:48:23.983476 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981501    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:48:23.983476 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981504    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:48:23.983476 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981508    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:48:23.983476 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981510    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:48:23.983476 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981513    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:48:23.983476 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981516    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:48:23.983476 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981519    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:48:23.983476 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981521    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:48:23.983476 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981524    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:48:23.983476 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981527    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:48:23.983476 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981529    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:48:23.983476 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981532    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:48:23.983476 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981534    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:48:23.983476 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981537    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:48:23.983476 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981539    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:48:23.983964 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981542    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:48:23.983964 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981545    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:48:23.983964 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981548    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:48:23.983964 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981550    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:48:23.983964 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981553    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:48:23.983964 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981556    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:48:23.983964 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981558    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:48:23.983964 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981561    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:48:23.983964 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981563    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:48:23.983964 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981566    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:48:23.983964 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981568    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:48:23.983964 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981571    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:48:23.983964 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981573    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:48:23.983964 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981576    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:48:23.983964 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981581    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:48:23.983964 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981583    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:48:23.983964 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981586    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:48:23.983964 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981589    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:48:23.983964 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981592    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:48:23.983964 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981594    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:48:23.984438 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981596    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:48:23.984438 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981599    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:48:23.984438 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981602    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:48:23.984438 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981605    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:48:23.984438 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981607    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:48:23.984438 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981610    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:48:23.984438 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981612    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:48:23.984438 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981615    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:48:23.984438 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981617    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:48:23.984438 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981620    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:48:23.984438 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981622    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:48:23.984438 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981624    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:48:23.984438 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981628    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:48:23.984438 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981632    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:48:23.984438 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981634    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:48:23.984438 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981637    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:48:23.984438 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981639    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:48:23.984438 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981642    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:48:23.984438 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981644    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:48:23.984438 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981646    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:48:23.984958 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981649    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:48:23.984958 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981651    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:48:23.984958 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981654    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:48:23.984958 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981656    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:48:23.984958 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981659    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:48:23.984958 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981661    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:48:23.984958 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981664    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:48:23.984958 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981667    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:48:23.984958 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981669    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:48:23.984958 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981672    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:48:23.984958 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981674    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:48:23.984958 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981677    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:48:23.984958 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981679    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:48:23.984958 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981682    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:48:23.984958 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.981686    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:48:23.984958 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982868    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:48:23.984958 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982875    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:48:23.984958 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982878    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:48:23.984958 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982881    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:48:23.985430 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982884    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:48:23.985430 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982887    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:48:23.985430 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982890    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:48:23.985430 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982893    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:48:23.985430 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982895    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:48:23.985430 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982898    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:48:23.985430 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982901    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:48:23.985430 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982903    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:48:23.985430 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982906    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:48:23.985430 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982909    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:48:23.985430 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982911    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:48:23.985430 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982914    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:48:23.985430 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982917    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:48:23.985430 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982921    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:48:23.985430 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982924    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:48:23.985430 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982927    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:48:23.985430 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982931    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:48:23.985430 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982933    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:48:23.985430 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982936    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:48:23.985895 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982946    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:48:23.985895 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982949    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:48:23.985895 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982952    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:48:23.985895 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982955    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:48:23.985895 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982957    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:48:23.985895 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982960    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:48:23.985895 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982962    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:48:23.985895 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982965    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:48:23.985895 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982968    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:48:23.985895 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982970    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:48:23.985895 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982973    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:48:23.985895 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982975    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:48:23.985895 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982978    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:48:23.985895 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982980    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:48:23.985895 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982982    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:48:23.985895 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982985    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:48:23.985895 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982987    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:48:23.985895 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982989    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:48:23.985895 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982992    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:48:23.985895 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982996    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:48:23.986404 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.982999    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:48:23.986404 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983002    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:48:23.986404 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983006    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:48:23.986404 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983009    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:48:23.986404 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983012    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:48:23.986404 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983014    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:48:23.986404 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983017    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:48:23.986404 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983019    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:48:23.986404 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983022    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:48:23.986404 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983024    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:48:23.986404 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983027    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:48:23.986404 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983029    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:48:23.986404 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983031    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:48:23.986404 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983034    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:48:23.986404 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983036    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:48:23.986404 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983039    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:48:23.986404 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983042    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:48:23.986404 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983045    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:48:23.986404 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983049    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:48:23.986917 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983052    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:48:23.986917 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983054    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:48:23.986917 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983057    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:48:23.986917 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983060    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:48:23.986917 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983062    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:48:23.986917 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983065    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:48:23.986917 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983067    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:48:23.986917 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983070    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:48:23.986917 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983072    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:48:23.986917 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983074    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:48:23.986917 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983077    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:48:23.986917 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983079    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:48:23.986917 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983082    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:48:23.986917 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983084    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:48:23.986917 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983086    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:48:23.986917 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983089    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:48:23.986917 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983092    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:48:23.986917 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983094    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:48:23.986917 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983097    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:48:23.986917 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983099    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:48:23.987412 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983102    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:48:23.987412 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983104    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:48:23.987412 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983107    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:48:23.987412 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983109    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:48:23.987412 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983111    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:48:23.987412 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983114    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:48:23.987412 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983116    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:48:23.987412 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983118    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:48:23.987412 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983121    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:48:23.987412 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983123    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:48:23.987412 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983126    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:48:23.987412 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983128    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:48:23.987412 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983130    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:48:23.987412 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983133    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:48:23.987412 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983136    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:48:23.987412 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983138    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:48:23.987412 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983141    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:48:23.987412 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983215    2559 flags.go:64] FLAG: --address="0.0.0.0"
Jul 31 14:48:23.987412 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983223    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 31 14:48:23.987412 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983229    2559 flags.go:64] FLAG: --anonymous-auth="true"
Jul 31 14:48:23.987412 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983234    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 31 14:48:23.987958 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983238    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 31 14:48:23.987958 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983241    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 31 14:48:23.987958 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983245    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 31 14:48:23.987958 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983249    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 31 14:48:23.987958 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983252    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 31 14:48:23.987958 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983255    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 31 14:48:23.987958 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983259    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 31 14:48:23.987958 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983262    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 31 14:48:23.987958 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983266    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 31 14:48:23.987958 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983269    2559 flags.go:64] FLAG: --cgroup-root=""
Jul 31 14:48:23.987958 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983272    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 31 14:48:23.987958 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983275    2559 flags.go:64] FLAG: --client-ca-file=""
Jul 31 14:48:23.987958 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983278    2559 flags.go:64] FLAG: --cloud-provider="external"
Jul 31 14:48:23.987958 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983280    2559 flags.go:64] FLAG: --cluster-dns="[]"
Jul 31 14:48:23.987958 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983285    2559 flags.go:64] FLAG: --cluster-domain=""
Jul 31 14:48:23.987958 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983288    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 31 14:48:23.987958 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983291    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 31 14:48:23.987958 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983293    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 31 14:48:23.987958 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983297    2559 flags.go:64] FLAG: --container-log-max-files="5"
Jul 31 14:48:23.987958 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983300    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 31 14:48:23.987958 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983303    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 31 14:48:23.987958 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983306    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 31 14:48:23.987958 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983309    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 31 14:48:23.988499 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983312    2559 flags.go:64] FLAG: --contention-profiling="false"
Jul 31 14:48:23.988499 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983315    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 31 14:48:23.988499 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983318    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 31 14:48:23.988499 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983321    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 31 14:48:23.988499 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983324    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 31 14:48:23.988499 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983328    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 31 14:48:23.988499 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983331    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 31 14:48:23.988499 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983334    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 31 14:48:23.988499 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983336    2559 flags.go:64] FLAG: --enable-load-reader="false"
Jul 31 14:48:23.988499 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983339    2559 flags.go:64] FLAG: --enable-server="true"
Jul 31 14:48:23.988499 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983342    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 31 14:48:23.988499 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983346    2559 flags.go:64] FLAG: --event-burst="100"
Jul 31 14:48:23.988499 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983349    2559 flags.go:64] FLAG: --event-qps="50"
Jul 31 14:48:23.988499 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983352    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 31 14:48:23.988499 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983354    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 31 14:48:23.988499 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983358    2559 flags.go:64] FLAG: --eviction-hard=""
Jul 31 14:48:23.988499 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983364    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 31 14:48:23.988499 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983367    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 31 14:48:23.988499 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983370    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 31 14:48:23.988499 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983373    2559 flags.go:64] FLAG: --eviction-soft=""
Jul 31 14:48:23.988499 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983376    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 31 14:48:23.988499 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983379    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 31 14:48:23.988499 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983382    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 31 14:48:23.988499 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983384    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 31 14:48:23.988499 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983387    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 31 14:48:23.989135 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983390    2559 flags.go:64] FLAG: --fail-swap-on="true"
Jul 31 14:48:23.989135 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983393    2559 flags.go:64] FLAG: --feature-gates=""
Jul 31 14:48:23.989135 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983397    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 31 14:48:23.989135 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983399    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 31 14:48:23.989135 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983402    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 31 14:48:23.989135 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983405    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 31 14:48:23.989135 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983408    2559 flags.go:64] FLAG: --healthz-port="10248"
Jul 31 14:48:23.989135 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983411    2559 flags.go:64] FLAG: --help="false"
Jul 31 14:48:23.989135 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983415    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-142-219.ec2.internal"
Jul 31 14:48:23.989135 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983419    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 31 14:48:23.989135 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983422    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 31 14:48:23.989135 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983425    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 31 14:48:23.989135 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983428    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 31 14:48:23.989135 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983432    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 31 14:48:23.989135 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983435    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 31 14:48:23.989135 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983437    2559 flags.go:64] FLAG: --image-service-endpoint=""
Jul 31 14:48:23.989135 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983440    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 31 14:48:23.989135 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983443    2559 flags.go:64] FLAG: --kube-api-burst="100"
Jul 31 14:48:23.989135 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983445    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 31 14:48:23.989135 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983449    2559 flags.go:64] FLAG: --kube-api-qps="50"
Jul 31 14:48:23.989135 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983451    2559 flags.go:64] FLAG: --kube-reserved=""
Jul 31 14:48:23.989135 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983454    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 31 14:48:23.989135 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983457    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 31 14:48:23.989135 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983461    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 31 14:48:23.989797 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983464    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 31 14:48:23.989797 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983467    2559 flags.go:64] FLAG: --lock-file=""
Jul 31 14:48:23.989797 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983470    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 31 14:48:23.989797 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983473    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 31 14:48:23.989797 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983476    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 31 14:48:23.989797 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983482    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 31 14:48:23.989797 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983484    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 31 14:48:23.989797 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983487    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 31 14:48:23.989797 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983490    2559 flags.go:64] FLAG: --logging-format="text"
Jul 31 14:48:23.989797 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983493    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 31 14:48:23.989797 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983496    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 31 14:48:23.989797 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983499    2559 flags.go:64] FLAG: --manifest-url=""
Jul 31 14:48:23.989797 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983501    2559 flags.go:64] FLAG: --manifest-url-header=""
Jul 31 14:48:23.989797 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983505    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 31 14:48:23.989797 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983508    2559 flags.go:64] FLAG: --max-open-files="1000000"
Jul 31 14:48:23.989797 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983513    2559 flags.go:64] FLAG: --max-pods="110"
Jul 31 14:48:23.989797 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983517    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 31 14:48:23.989797 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983520    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 31 14:48:23.989797 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983523    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 31 14:48:23.989797 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983525    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 31 14:48:23.989797 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983528    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 31 14:48:23.989797 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983531    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 31 14:48:23.989797 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983534    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 31 14:48:23.989797 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983541    2559 flags.go:64] FLAG: --node-status-max-images="50"
Jul 31 14:48:23.989797 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983544    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 31 14:48:23.990413 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983547    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 31 14:48:23.990413 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983550    2559 flags.go:64] FLAG: --pod-cidr=""
Jul 31 14:48:23.990413 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983553    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Jul 31 14:48:23.990413 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983558    2559 flags.go:64] FLAG: --pod-manifest-path=""
Jul 31 14:48:23.990413 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983561    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 31 14:48:23.990413 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983565    2559 flags.go:64] FLAG: --pods-per-core="0"
Jul 31 14:48:23.990413 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983569    2559 flags.go:64] FLAG: --port="10250"
Jul 31 14:48:23.990413 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983572    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 31 14:48:23.990413 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983575    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07b6d2f27061dd6e0"
Jul 31 14:48:23.990413 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983578    2559 flags.go:64] FLAG: --qos-reserved=""
Jul 31 14:48:23.990413 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983582    2559 flags.go:64] FLAG: --read-only-port="10255"
Jul 31 14:48:23.990413 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983585    2559 flags.go:64] FLAG: --register-node="true"
Jul 31 14:48:23.990413 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983588    2559 flags.go:64] FLAG: --register-with-taints=""
Jul 31 14:48:23.990413 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983592    2559 flags.go:64] FLAG: --registry-burst="10"
Jul 31 14:48:23.990413 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983595    2559 flags.go:64] FLAG: --registry-qps="5"
Jul 31 14:48:23.990413 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983597    2559 flags.go:64] FLAG: --reserved-cpus=""
Jul 31 14:48:23.990413 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983600    2559 flags.go:64] FLAG: --reserved-memory=""
Jul 31 14:48:23.990413 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983604    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 31 14:48:23.990413 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983607    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 31 14:48:23.990413 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983610    2559 flags.go:64] FLAG: --rotate-certificates="false"
Jul 31 14:48:23.990413 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983612    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 31 14:48:23.990413 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983615    2559 flags.go:64] FLAG: --runonce="false"
Jul 31 14:48:23.990413 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983618    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 31 14:48:23.990413 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983621    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 31 14:48:23.990413 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983625    2559 flags.go:64] FLAG: --seccomp-default="false"
Jul 31 14:48:23.991300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983627    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 31 14:48:23.991300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983630    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 31 14:48:23.991300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983633    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 31 14:48:23.991300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983636    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 31 14:48:23.991300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983639    2559 flags.go:64] FLAG: --storage-driver-password="root"
Jul 31 14:48:23.991300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983642    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 31 14:48:23.991300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983645    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 31 14:48:23.991300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983648    2559 flags.go:64] FLAG: --storage-driver-user="root"
Jul 31 14:48:23.991300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983650    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 31 14:48:23.991300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983653    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 31 14:48:23.991300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983656    2559 flags.go:64] FLAG: --system-cgroups=""
Jul 31 14:48:23.991300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983658    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 31 14:48:23.991300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983663    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 31 14:48:23.991300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983668    2559 flags.go:64] FLAG: --tls-cert-file=""
Jul 31 14:48:23.991300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983671    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 31 14:48:23.991300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983678    2559 flags.go:64] FLAG: --tls-min-version=""
Jul 31 14:48:23.991300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983680    2559 flags.go:64] FLAG: --tls-private-key-file=""
Jul 31 14:48:23.991300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983683    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 31 14:48:23.991300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983686    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 31 14:48:23.991300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983689    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 31 14:48:23.991300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983692    2559 flags.go:64] FLAG: --v="2"
Jul 31 14:48:23.991300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983696    2559 flags.go:64] FLAG: --version="false"
Jul 31 14:48:23.991300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983700    2559 flags.go:64] FLAG: --vmodule=""
Jul 31 14:48:23.991300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983703    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 31 14:48:23.991300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.983706    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 31 14:48:23.992054 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983838    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:48:23.992054 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983842    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:48:23.992054 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983845    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:48:23.992054 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983848    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:48:23.992054 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983851    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:48:23.992054 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983869    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:48:23.992054 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983875    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:48:23.992054 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983878    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:48:23.992054 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983881    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:48:23.992054 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983883    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:48:23.992054 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983886    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:48:23.992054 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983888    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:48:23.992054 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983891    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:48:23.992054 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983893    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:48:23.992054 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983896    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:48:23.992054 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983898    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:48:23.992054 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983901    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:48:23.992054 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983903    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:48:23.992054 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983906    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:48:23.992054 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983915    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:48:23.992561 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983920    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:48:23.992561 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983923    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:48:23.992561 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983926    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:48:23.992561 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983929    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:48:23.992561 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983931    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:48:23.992561 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983934    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:48:23.992561 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983937    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:48:23.992561 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983940    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:48:23.992561 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983943    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:48:23.992561 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983945    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:48:23.992561 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983949    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:48:23.992561 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983952    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:48:23.992561 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983955    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:48:23.992561 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983958    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:48:23.992561 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983961    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:48:23.992561 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983963    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:48:23.992561 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983966    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:48:23.992561 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983969    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:48:23.992561 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983973    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:48:23.993046 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983976    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:48:23.993046 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983979    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:48:23.993046 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983982    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:48:23.993046 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983984    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:48:23.993046 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983987    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:48:23.993046 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983989    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:48:23.993046 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983992    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:48:23.993046 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983994    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:48:23.993046 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983997    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:48:23.993046 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.983999    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:48:23.993046 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984002    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:48:23.993046 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984005    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:48:23.993046 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984007    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:48:23.993046 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984011    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:48:23.993046 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984013    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:48:23.993046 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984016    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:48:23.993046 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984018    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:48:23.993046 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984020    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:48:23.993046 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984023    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:48:23.993046 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984026    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:48:23.993529 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984028    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:48:23.993529 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984031    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:48:23.993529 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984033    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:48:23.993529 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984036    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:48:23.993529 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984038    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:48:23.993529 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984040    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:48:23.993529 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984043    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:48:23.993529 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984045    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:48:23.993529 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984048    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:48:23.993529 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984052    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:48:23.993529 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984055    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:48:23.993529 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984059    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:48:23.993529 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984062    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:48:23.993529 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984064    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:48:23.993529 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984067    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:48:23.993529 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984069    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:48:23.993529 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984072    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:48:23.993529 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984074    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:48:23.993529 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984077    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:48:23.994024 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984079    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:48:23.994024 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984082    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:48:23.994024 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984084    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:48:23.994024 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984087    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:48:23.994024 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984089    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:48:23.994024 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984092    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:48:23.994024 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984095    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:48:23.994024 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984098    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:48:23.994024 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984101    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:48:23.994024 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984103    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:48:23.994024 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984106    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:48:23.994024 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984108    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:48:23.994024 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984112    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:48:23.994024 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984115    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:48:23.994024 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984118    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:48:23.994024 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984120    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:48:23.994024 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984123    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:48:23.994024 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984125    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:48:23.994024 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984127    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:48:23.994024 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984130    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:48:23.994509 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.984132    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:48:23.994509 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.984137    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 14:48:23.994509 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.993426    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 31 14:48:23.994509 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.993443    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 31 14:48:23.994509 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.994101    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 31 14:48:23.994509 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.994110    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 31 14:48:23.994509 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994175    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:48:23.994509 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994181    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:48:23.994509 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994185    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:48:23.994509 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994189    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:48:23.994509 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994192    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:48:23.994509 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994195    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:48:23.994509 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994198    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:48:23.994509 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994201    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:48:23.994875 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994203    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:48:23.994875 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994206    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:48:23.994875 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994209    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:48:23.994875 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994212    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:48:23.994875 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994214    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:48:23.994875 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994218    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:48:23.994875 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994221    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:48:23.994875 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994224    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:48:23.994875 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994226    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:48:23.994875 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994229    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:48:23.994875 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994232    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:48:23.994875 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994235    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:48:23.994875 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994239    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:48:23.994875 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994243    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:48:23.994875 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994245    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:48:23.994875 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994248    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:48:23.994875 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994250    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:48:23.994875 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994253    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:48:23.994875 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994255    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:48:23.995394 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994258    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:48:23.995394 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994261    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:48:23.995394 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994263    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:48:23.995394 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994266    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:48:23.995394 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994268    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:48:23.995394 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994271    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:48:23.995394 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994273    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:48:23.995394 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994278    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:48:23.995394 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994282    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:48:23.995394 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994286    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:48:23.995394 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994289    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:48:23.995394 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994293    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:48:23.995394 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994295    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:48:23.995394 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994298    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:48:23.995394 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994301    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:48:23.995394 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994304    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:48:23.995394 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994306    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:48:23.995394 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994309    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:48:23.995394 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994312    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:48:23.995873 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994315    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:48:23.995873 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994317    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:48:23.995873 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994320    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:48:23.995873 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994323    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:48:23.995873 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994325    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:48:23.995873 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994328    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:48:23.995873 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994330    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:48:23.995873 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994333    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:48:23.995873 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994335    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:48:23.995873 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994337    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:48:23.995873 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994340    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:48:23.995873 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994343    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:48:23.995873 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994345    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:48:23.995873 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994348    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:48:23.995873 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994351    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:48:23.995873 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994354    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:48:23.995873 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994356    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:48:23.995873 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994359    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:48:23.995873 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994362    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:48:23.995873 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994364    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:48:23.996361 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994367    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:48:23.996361 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994369    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:48:23.996361 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994372    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:48:23.996361 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994374    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:48:23.996361 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994377    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:48:23.996361 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994379    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:48:23.996361 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994381    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:48:23.996361 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994384    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:48:23.996361 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994387    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:48:23.996361 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994389    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:48:23.996361 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994391    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:48:23.996361 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994394    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:48:23.996361 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994396    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:48:23.996361 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994400    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:48:23.996361 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994403    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:48:23.996361 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994406    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:48:23.996361 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994408    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:48:23.996361 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994411    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:48:23.996361 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994413    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:48:23.996361 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994416    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:48:23.996869 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994418    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:48:23.996869 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994420    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:48:23.996869 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994423    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:48:23.996869 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994425    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:48:23.996869 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994428    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:48:23.996869 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994431    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:48:23.996869 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994433    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:48:23.996869 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994436    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:48:23.996869 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994438    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:48:23.996869 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994441    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:48:23.996869 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994443    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:48:23.996869 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994446    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:48:23.996869 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994448    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:48:23.996869 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.994454    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 14:48:23.996869 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994580    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:48:23.997236 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994586    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:48:23.997236 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994589    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:48:23.997236 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994592    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:48:23.997236 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994594    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:48:23.997236 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994597    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:48:23.997236 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994600    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:48:23.997236 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994603    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:48:23.997236 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994606    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:48:23.997236 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994608    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:48:23.997236 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994611    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:48:23.997236 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994613    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:48:23.997236 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994617    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:48:23.997236 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994620    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:48:23.997236 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994622    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:48:23.997236 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994625    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:48:23.997236 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994627    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:48:23.997236 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994629    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:48:23.997236 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994632    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:48:23.997236 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994634    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:48:23.997700 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994637    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:48:23.997700 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994640    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:48:23.997700 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994642    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:48:23.997700 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994645    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:48:23.997700 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994647    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:48:23.997700 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994650    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:48:23.997700 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994653    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:48:23.997700 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994656    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:48:23.997700 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994659    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:48:23.997700 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994661    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:48:23.997700 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994664    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:48:23.997700 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994666    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:48:23.997700 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994668    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:48:23.997700 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994671    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:48:23.997700 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994673    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:48:23.997700 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994676    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:48:23.997700 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994678    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:48:23.997700 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994681    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:48:23.997700 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994683    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:48:23.997700 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994685    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:48:23.998202 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994688    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:48:23.998202 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994690    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:48:23.998202 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994693    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:48:23.998202 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994695    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:48:23.998202 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994697    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:48:23.998202 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994700    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:48:23.998202 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994703    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:48:23.998202 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994706    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:48:23.998202 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994708    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:48:23.998202 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994711    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:48:23.998202 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994713    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:48:23.998202 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994715    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:48:23.998202 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994718    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:48:23.998202 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994720    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:48:23.998202 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994723    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:48:23.998202 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994741    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:48:23.998202 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994745    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:48:23.998202 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994749    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:48:23.998202 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994751    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:48:23.998202 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994754    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:48:23.998705 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994756    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:48:23.998705 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994759    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:48:23.998705 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994761    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:48:23.998705 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994764    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:48:23.998705 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994766    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:48:23.998705 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994769    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:48:23.998705 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994771    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:48:23.998705 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994774    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:48:23.998705 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994776    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:48:23.998705 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994779    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:48:23.998705 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994781    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:48:23.998705 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994784    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:48:23.998705 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994786    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:48:23.998705 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994790    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:48:23.998705 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994793    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:48:23.998705 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994796    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:48:23.998705 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994799    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:48:23.998705 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994802    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:48:23.998705 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994806    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:48:23.999299 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994808    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:48:23.999299 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994811    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:48:23.999299 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994813    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:48:23.999299 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994816    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:48:23.999299 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994818    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:48:23.999299 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994820    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:48:23.999299 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994823    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:48:23.999299 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994825    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:48:23.999299 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994828    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:48:23.999299 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994830    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:48:23.999299 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994835    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:48:23.999299 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994838    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:48:23.999299 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994841    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:48:23.999299 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994843    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:48:23.999299 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994846    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:48:23.999299 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994848    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:48:23.999299 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994851    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:48:23.999299 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994853    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:48:23.999299 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994855    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:48:23.999299 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:23.994858    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:48:23.999791 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.994863    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 14:48:23.999791 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.994989    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 31 14:48:23.999791 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.998090    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 31 14:48:23.999791 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.999499    2559 server.go:1019] "Starting client certificate rotation"
Jul 31 14:48:23.999791 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.999600    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 14:48:23.999791 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:23.999631    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 14:48:24.001999 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.001984    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 14:48:24.006271 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.006255    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 14:48:24.016183 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.016159    2559 log.go:25] "Validated CRI v1 runtime API"
Jul 31 14:48:24.018233 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.018216    2559 log.go:25] "Validated CRI v1 image API"
Jul 31 14:48:24.019067 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.019052    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 31 14:48:24.022369 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.022348    2559 fs.go:135] Filesystem UUIDs: map[52793155-2b43-4871-81bf-818213e8dc38:/dev/nvme0n1p3 55154263-0d1e-4295-960e-90e7d8ab8f84:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 31 14:48:24.022452 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.022369    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 31 14:48:24.027411 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.027296    2559 manager.go:216] Machine: {Timestamp:2026-07-31 14:48:24.026014195 +0000 UTC m=+0.337571827 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099689 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21e14bc37b92775b40f02c4e15d24e SystemUUID:ec21e14b-c37b-9277-5b40-f02c4e15d24e BootID:9074b0ff-02cd-4890-8b5d-60d43d163f91 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:85:c5:3c:69:97 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:85:c5:3c:69:97 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:c1:26:11:c9:19 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 31 14:48:24.027411 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.027399    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 31 14:48:24.027565 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.027516    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 31 14:48:24.028098 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.028082    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 14:48:24.031295 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.031266    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 31 14:48:24.031450 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.031298    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-219.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 31 14:48:24.032021 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.032010    2559 topology_manager.go:138] "Creating topology manager with none policy"
Jul 31 14:48:24.032053 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.032024    2559 container_manager_linux.go:309] "Creating device plugin manager"
Jul 31 14:48:24.032053 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.032033    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 14:48:24.032652 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.032641    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 14:48:24.033545 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.033535    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 31 14:48:24.037100 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.037088    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 14:48:24.037391 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.037379    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 31 14:48:24.037586 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.037564    2559 kubelet.go:486] "Attempting to sync node with API server"
Jul 31 14:48:24.037659 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.037593    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 31 14:48:24.037659 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.037613    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 31 14:48:24.037659 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.037630    2559 kubelet.go:398] "Adding apiserver pod source"
Jul 31 14:48:24.037659 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.037644    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 31 14:48:24.038185 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.038166    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 14:48:24.038322 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.038312    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 14:48:24.040933 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.040914    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 31 14:48:24.042633 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.042620    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 31 14:48:24.042672 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.042651    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 31 14:48:24.043744 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.043720    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 31 14:48:24.043780 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.043753    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 31 14:48:24.043780 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.043761    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 31 14:48:24.043780 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.043766    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 31 14:48:24.043780 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.043775    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 31 14:48:24.043780 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.043781    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 31 14:48:24.043917 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.043790    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 31 14:48:24.043917 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.043796    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 31 14:48:24.043917 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.043803    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 31 14:48:24.043917 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.043809    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 31 14:48:24.043917 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.043817    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 31 14:48:24.043917 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.043822    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 31 14:48:24.045220 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.045210    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 31 14:48:24.045220 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.045220    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 31 14:48:24.048690 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.048678    2559 server.go:1268] "Started kubelet"
Jul 31 14:48:24.048822 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.048802    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-219.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 31 14:48:24.048895 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.048807    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-219.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 31 14:48:24.048947 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.048882    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 31 14:48:24.048989 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.048886    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 31 14:48:24.048989 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.048964    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 31 14:48:24.049204 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.049173    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 31 14:48:24.049750 ip-10-0-142-219 systemd[1]: Started Kubernetes Kubelet.
Jul 31 14:48:24.051096 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.051081    2559 server.go:310] "Adding debug handlers to kubelet server"
Jul 31 14:48:24.052876 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.052860    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 31 14:48:24.056226 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.056204    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 31 14:48:24.057862 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.057845    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 31 14:48:24.057862 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.057855    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 31 14:48:24.058264 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.058247    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 31 14:48:24.058642 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.058625    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 31 14:48:24.058642 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.058627    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 31 14:48:24.058797 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.058649    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 31 14:48:24.058797 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.058724    2559 reconstruct.go:97] "Volume reconstruction finished"
Jul 31 14:48:24.058797 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.058753    2559 reconciler.go:29] "Reconciler: start to sync state"
Jul 31 14:48:24.058969 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.058840    2559 factory.go:55] Registering systemd factory
Jul 31 14:48:24.058969 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.058876    2559 factory.go:223] Registration of the systemd container factory successfully
Jul 31 14:48:24.059153 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.059134    2559 factory.go:153] Registering CRI-O factory
Jul 31 14:48:24.059153 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.059154    2559 factory.go:223] Registration of the crio container factory successfully
Jul 31 14:48:24.059297 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.059206    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 31 14:48:24.059297 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.059229    2559 factory.go:103] Registering Raw factory
Jul 31 14:48:24.059297 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.059245    2559 manager.go:1196] Started watching for new ooms in manager
Jul 31 14:48:24.059425 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.058096    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-219.ec2.internal.18c7674082f993ea  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-219.ec2.internal,UID:ip-10-0-142-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-219.ec2.internal,},FirstTimestamp:2026-07-31 14:48:24.04865329 +0000 UTC m=+0.360210907,LastTimestamp:2026-07-31 14:48:24.04865329 +0000 UTC m=+0.360210907,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-219.ec2.internal,}"
Jul 31 14:48:24.059994 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.059980    2559 manager.go:349] Starting recovery of all containers
Jul 31 14:48:24.064991 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.064958    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 31 14:48:24.065198 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.065166    2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-219.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 31 14:48:24.071470 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.071438    2559 manager.go:354] Recovery completed
Jul 31 14:48:24.073071 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.072934    2559 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 31 14:48:24.076246 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.076229    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:24.078123 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.078098    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:24.078210 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.078141    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:24.078210 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.078158    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:24.078639 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.078625    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 31 14:48:24.078639 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.078635    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:24.078639 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.078639    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 31 14:48:24.078811 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.078657    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 14:48:24.079404 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.079387    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:24.079483 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.079418    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:24.079483 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.079433    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:24.079997 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.079983    2559 policy_none.go:49] "None policy: Start"
Jul 31 14:48:24.080051 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.079945    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-219.ec2.internal.18c7674084bb44b7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-219.ec2.internal,UID:ip-10-0-142-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-219.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-219.ec2.internal,},FirstTimestamp:2026-07-31 14:48:24.078124215 +0000 UTC m=+0.389681831,LastTimestamp:2026-07-31 14:48:24.078124215 +0000 UTC m=+0.389681831,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-219.ec2.internal,}"
Jul 31 14:48:24.080092 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.080003    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 31 14:48:24.080119 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.080090    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 31 14:48:24.081109 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.081098    2559 policy_none.go:47] "Start"
Jul 31 14:48:24.089120 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.089048    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-219.ec2.internal.18c7674084bba29e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-219.ec2.internal,UID:ip-10-0-142-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-219.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-219.ec2.internal,},FirstTimestamp:2026-07-31 14:48:24.078148254 +0000 UTC m=+0.389705868,LastTimestamp:2026-07-31 14:48:24.078148254 +0000 UTC m=+0.389705868,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-219.ec2.internal,}"
Jul 31 14:48:24.091078 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.091017    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-219.ec2.internal.18c7674084bbe0ee  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-219.ec2.internal,UID:ip-10-0-142-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-219.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-219.ec2.internal,},FirstTimestamp:2026-07-31 14:48:24.078164206 +0000 UTC m=+0.389721821,LastTimestamp:2026-07-31 14:48:24.078164206 +0000 UTC m=+0.389721821,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-219.ec2.internal,}"
Jul 31 14:48:24.101500 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.101436    2559 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-219.ec2.internal.18c7674084bb44b7\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-219.ec2.internal.18c7674084bb44b7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-219.ec2.internal,UID:ip-10-0-142-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-219.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-219.ec2.internal,},FirstTimestamp:2026-07-31 14:48:24.078124215 +0000 UTC m=+0.389681831,LastTimestamp:2026-07-31 14:48:24.079402134 +0000 UTC m=+0.390959751,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-219.ec2.internal,}"
Jul 31 14:48:24.111301 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.111226    2559 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-219.ec2.internal.18c7674084bba29e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-219.ec2.internal.18c7674084bba29e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-219.ec2.internal,UID:ip-10-0-142-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-219.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-219.ec2.internal,},FirstTimestamp:2026-07-31 14:48:24.078148254 +0000 UTC m=+0.389705868,LastTimestamp:2026-07-31 14:48:24.079424973 +0000 UTC m=+0.390982587,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-219.ec2.internal,}"
Jul 31 14:48:24.120993 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.120924    2559 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-219.ec2.internal.18c7674084bbe0ee\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-219.ec2.internal.18c7674084bbe0ee  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-219.ec2.internal,UID:ip-10-0-142-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-219.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-219.ec2.internal,},FirstTimestamp:2026-07-31 14:48:24.078164206 +0000 UTC m=+0.389721821,LastTimestamp:2026-07-31 14:48:24.079438601 +0000 UTC m=+0.390996216,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-219.ec2.internal,}"
Jul 31 14:48:24.134487 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.121413    2559 manager.go:337] "Starting Device Plugin manager"
Jul 31 14:48:24.134487 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.121442    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 31 14:48:24.134487 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.121452    2559 server.go:87] "Starting device plugin registration server"
Jul 31 14:48:24.134487 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.121690    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 31 14:48:24.134487 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.121701    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 31 14:48:24.134487 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.121787    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 31 14:48:24.134487 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.121885    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 31 14:48:24.134487 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.121894    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 31 14:48:24.134487 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.122388    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 31 14:48:24.134487 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.122442    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 31 14:48:24.134487 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.131889    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-219.ec2.internal.18c767408774b506  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-219.ec2.internal,UID:ip-10-0-142-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-142-219.ec2.internal,},FirstTimestamp:2026-07-31 14:48:24.123831558 +0000 UTC m=+0.435389158,LastTimestamp:2026-07-31 14:48:24.123831558 +0000 UTC m=+0.435389158,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-219.ec2.internal,}"
Jul 31 14:48:24.140716 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.140696    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b98kv"
Jul 31 14:48:24.148706 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.148686    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b98kv"
Jul 31 14:48:24.214055 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.213978    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 31 14:48:24.215395 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.215371    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 31 14:48:24.215504 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.215413    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 31 14:48:24.215504 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.215436    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 31 14:48:24.215504 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.215475    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 31 14:48:24.222626 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.222611    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:24.223431 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.223414    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:24.223526 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.223443    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:24.223526 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.223453    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:24.223526 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.223481    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-219.ec2.internal"
Jul 31 14:48:24.230028 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.230012    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:48:24.235959 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.235942    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-219.ec2.internal"
Jul 31 14:48:24.236041 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.235966    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-219.ec2.internal\": node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 31 14:48:24.249680 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.249655    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 31 14:48:24.315751 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.315684    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-219.ec2.internal"]
Jul 31 14:48:24.315929 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.315846    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:24.316854 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.316840    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:24.316921 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.316872    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:24.316921 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.316886    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:24.318471 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.318458    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:24.318601 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.318587    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal"
Jul 31 14:48:24.318666 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.318623    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:24.319170 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.319147    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:24.319256 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.319181    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:24.319256 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.319195    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:24.319256 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.319196    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:24.319256 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.319215    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:24.319256 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.319224    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:24.320379 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.320364    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-219.ec2.internal"
Jul 31 14:48:24.320428 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.320397    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:24.321030 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.321012    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:24.321119 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.321047    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:24.321119 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.321062    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:24.344302 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.344281    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-219.ec2.internal\" not found" node="ip-10-0-142-219.ec2.internal"
Jul 31 14:48:24.348698 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.348684    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-219.ec2.internal\" not found" node="ip-10-0-142-219.ec2.internal"
Jul 31 14:48:24.350740 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.350713    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 31 14:48:24.360349 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.360335    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/13814aefbbec7fd254d31d3cf317ab12-config\") pod \"kube-apiserver-proxy-ip-10-0-142-219.ec2.internal\" (UID: \"13814aefbbec7fd254d31d3cf317ab12\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-219.ec2.internal"
Jul 31 14:48:24.360406 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.360357    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87bd827f8cab5b15b18522f993681f04-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal\" (UID: \"87bd827f8cab5b15b18522f993681f04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal"
Jul 31 14:48:24.360406 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.360378    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87bd827f8cab5b15b18522f993681f04-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal\" (UID: \"87bd827f8cab5b15b18522f993681f04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal"
Jul 31 14:48:24.451328 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.451286    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 31 14:48:24.460910 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.460881    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/13814aefbbec7fd254d31d3cf317ab12-config\") pod \"kube-apiserver-proxy-ip-10-0-142-219.ec2.internal\" (UID: \"13814aefbbec7fd254d31d3cf317ab12\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-219.ec2.internal"
Jul 31 14:48:24.460910 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.460912    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87bd827f8cab5b15b18522f993681f04-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal\" (UID: \"87bd827f8cab5b15b18522f993681f04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal"
Jul 31 14:48:24.461091 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.460930    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87bd827f8cab5b15b18522f993681f04-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal\" (UID: \"87bd827f8cab5b15b18522f993681f04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal"
Jul 31 14:48:24.461091 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.460990    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87bd827f8cab5b15b18522f993681f04-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal\" (UID: \"87bd827f8cab5b15b18522f993681f04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal"
Jul 31 14:48:24.461091 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.461011    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/13814aefbbec7fd254d31d3cf317ab12-config\") pod \"kube-apiserver-proxy-ip-10-0-142-219.ec2.internal\" (UID: \"13814aefbbec7fd254d31d3cf317ab12\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-219.ec2.internal"
Jul 31 14:48:24.461091 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.461038    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87bd827f8cab5b15b18522f993681f04-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal\" (UID: \"87bd827f8cab5b15b18522f993681f04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal"
Jul 31 14:48:24.551988 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.551922    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 31 14:48:24.648461 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.648434    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal"
Jul 31 14:48:24.651339 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.651323    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-219.ec2.internal"
Jul 31 14:48:24.653044 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.652771    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 31 14:48:24.753699 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.753669    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 31 14:48:24.854219 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.854124    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 31 14:48:24.954653 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:24.954622    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 31 14:48:24.999132 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.999099    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 31 14:48:24.999562 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:24.999259    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 31 14:48:25.054842 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:25.054806    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 31 14:48:25.153336 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:25.153286    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-30 14:43:24 +0000 UTC" deadline="2027-12-24 21:24:46.507828347 +0000 UTC"
Jul 31 14:48:25.153336 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:25.153332    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12270h36m21.354499423s"
Jul 31 14:48:25.155449 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:25.155425    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 31 14:48:25.204383 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:25.204351    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13814aefbbec7fd254d31d3cf317ab12.slice/crio-ac037ca226e578b0398972d8f27c154bdaf3e52973670e2670f6a35788809152 WatchSource:0}: Error finding container ac037ca226e578b0398972d8f27c154bdaf3e52973670e2670f6a35788809152: Status 404 returned error can't find the container with id ac037ca226e578b0398972d8f27c154bdaf3e52973670e2670f6a35788809152
Jul 31 14:48:25.207431 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:25.207414    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 14:48:25.218764 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:25.218703    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-219.ec2.internal" event={"ID":"13814aefbbec7fd254d31d3cf317ab12","Type":"ContainerStarted","Data":"ac037ca226e578b0398972d8f27c154bdaf3e52973670e2670f6a35788809152"}
Jul 31 14:48:25.255879 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:25.255845    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-219.ec2.internal\" not found"
Jul 31 14:48:25.286306 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:25.286287    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:25.287425 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:25.287411    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:25.287489 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:25.287440    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:25.287489 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:25.287450    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:25.288446 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:25.288431    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:48:25.358699 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:25.358634    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal"
Jul 31 14:48:25.370303 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:25.370285    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 14:48:25.371810 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:25.371797    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-219.ec2.internal"
Jul 31 14:48:25.384835 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:25.384814    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 14:48:25.439028 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:25.439006    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:48:25.477036 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:25.477008    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:48:25.818758 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:25.818664    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:48:26.040565 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.040533    2559 apiserver.go:52] "Watching apiserver"
Jul 31 14:48:26.047165 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.047141    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 31 14:48:26.047578 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.047549    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-jfrf7","kube-system/global-pull-secret-syncer-nk8pw","openshift-dns/node-resolver-mpkxl","openshift-image-registry/node-ca-bwcjx","openshift-multus/network-metrics-daemon-b45r4","openshift-network-diagnostics/network-check-target-6vfsr","openshift-network-operator/iptables-alerter-tcn8w","openshift-ovn-kubernetes/ovnkube-node-qp8cd","kube-system/konnectivity-agent-xg6w7","kube-system/kube-apiserver-proxy-ip-10-0-142-219.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57","openshift-cluster-node-tuning-operator/tuned-xvpvq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal","openshift-multus/multus-64wqp"]
Jul 31 14:48:26.050368 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.050344    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tcn8w"
Jul 31 14:48:26.052752 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.052715    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57"
Jul 31 14:48:26.054832 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.054801    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mpkxl"
Jul 31 14:48:26.057133 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.057031    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bwcjx"
Jul 31 14:48:26.059460 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.059107    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 31 14:48:26.059460 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.059300    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:26.059460 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:26.059369    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b45r4" podUID="355d9cbb-1867-49cc-8ec2-564b4aa6daea"
Jul 31 14:48:26.061589 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.061457    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:48:26.061589 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:26.061526    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vfsr" podUID="567a8972-6aaf-499f-a8d8-a83da117da60"
Jul 31 14:48:26.061786 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.061648    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 31 14:48:26.061907 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.061884    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 31 14:48:26.062012 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.061909    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:48:26.062012 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.061988    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 31 14:48:26.062122 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.061892    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7fcf7\""
Jul 31 14:48:26.065560 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.062311    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 31 14:48:26.065560 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.063183    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 31 14:48:26.065560 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.063220    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 31 14:48:26.065560 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.063336    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 31 14:48:26.065560 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.063461    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wfkfx\""
Jul 31 14:48:26.065560 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.063640    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 31 14:48:26.065560 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.063720    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-s4qdc\""
Jul 31 14:48:26.065560 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.064033    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 31 14:48:26.065560 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.064145    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 31 14:48:26.065560 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.064470    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rr9h5\""
Jul 31 14:48:26.066365 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.066339    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xg6w7"
Jul 31 14:48:26.068467 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.068446    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhqsp\" (UniqueName: \"kubernetes.io/projected/16b5669e-0dde-49c0-854b-495854e99a82-kube-api-access-dhqsp\") pod \"node-ca-bwcjx\" (UID: \"16b5669e-0dde-49c0-854b-495854e99a82\") " pod="openshift-image-registry/node-ca-bwcjx"
Jul 31 14:48:26.068551 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.068478    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a734dd47-fffc-4dfd-8946-3c3466565b8c-host-slash\") pod \"iptables-alerter-tcn8w\" (UID: \"a734dd47-fffc-4dfd-8946-3c3466565b8c\") " pod="openshift-network-operator/iptables-alerter-tcn8w"
Jul 31 14:48:26.068551 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.068500    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4c2a7673-4169-4edb-a234-70c7d12e0804-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wmw57\" (UID: \"4c2a7673-4169-4edb-a234-70c7d12e0804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57"
Jul 31 14:48:26.068551 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.068516    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4c2a7673-4169-4edb-a234-70c7d12e0804-device-dir\") pod \"aws-ebs-csi-driver-node-wmw57\" (UID: \"4c2a7673-4169-4edb-a234-70c7d12e0804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57"
Jul 31 14:48:26.068551 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.068532    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqpwq\" (UniqueName: \"kubernetes.io/projected/54adc061-d1ce-4ba7-83cf-3b2968693d52-kube-api-access-sqpwq\") pod \"node-resolver-mpkxl\" (UID: \"54adc061-d1ce-4ba7-83cf-3b2968693d52\") " pod="openshift-dns/node-resolver-mpkxl"
Jul 31 14:48:26.068551 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.068548    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16b5669e-0dde-49c0-854b-495854e99a82-host\") pod \"node-ca-bwcjx\" (UID: \"16b5669e-0dde-49c0-854b-495854e99a82\") " pod="openshift-image-registry/node-ca-bwcjx"
Jul 31 14:48:26.068784 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.068572    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a734dd47-fffc-4dfd-8946-3c3466565b8c-iptables-alerter-script\") pod \"iptables-alerter-tcn8w\" (UID: \"a734dd47-fffc-4dfd-8946-3c3466565b8c\") " pod="openshift-network-operator/iptables-alerter-tcn8w"
Jul 31 14:48:26.068784 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.068622    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4c2a7673-4169-4edb-a234-70c7d12e0804-socket-dir\") pod \"aws-ebs-csi-driver-node-wmw57\" (UID: \"4c2a7673-4169-4edb-a234-70c7d12e0804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57"
Jul 31 14:48:26.068784 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.068645    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/16b5669e-0dde-49c0-854b-495854e99a82-serviceca\") pod \"node-ca-bwcjx\" (UID: \"16b5669e-0dde-49c0-854b-495854e99a82\") " pod="openshift-image-registry/node-ca-bwcjx"
Jul 31 14:48:26.068784 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.068684    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355d9cbb-1867-49cc-8ec2-564b4aa6daea-metrics-certs\") pod \"network-metrics-daemon-b45r4\" (UID: \"355d9cbb-1867-49cc-8ec2-564b4aa6daea\") " pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:26.068784 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.068701    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zshq2\" (UniqueName: \"kubernetes.io/projected/355d9cbb-1867-49cc-8ec2-564b4aa6daea-kube-api-access-zshq2\") pod \"network-metrics-daemon-b45r4\" (UID: \"355d9cbb-1867-49cc-8ec2-564b4aa6daea\") " pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:26.068784 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.068716    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/54adc061-d1ce-4ba7-83cf-3b2968693d52-hosts-file\") pod \"node-resolver-mpkxl\" (UID: \"54adc061-d1ce-4ba7-83cf-3b2968693d52\") " pod="openshift-dns/node-resolver-mpkxl"
Jul 31 14:48:26.068784 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.068731    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4c2a7673-4169-4edb-a234-70c7d12e0804-etc-selinux\") pod \"aws-ebs-csi-driver-node-wmw57\" (UID: \"4c2a7673-4169-4edb-a234-70c7d12e0804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57"
Jul 31 14:48:26.068784 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.068750    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4c2a7673-4169-4edb-a234-70c7d12e0804-sys-fs\") pod \"aws-ebs-csi-driver-node-wmw57\" (UID: \"4c2a7673-4169-4edb-a234-70c7d12e0804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57"
Jul 31 14:48:26.069114 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.068791    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb7ng\" (UniqueName: \"kubernetes.io/projected/4c2a7673-4169-4edb-a234-70c7d12e0804-kube-api-access-nb7ng\") pod \"aws-ebs-csi-driver-node-wmw57\" (UID: \"4c2a7673-4169-4edb-a234-70c7d12e0804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57"
Jul 31 14:48:26.069114 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.068813    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9bd8\" (UniqueName: \"kubernetes.io/projected/a734dd47-fffc-4dfd-8946-3c3466565b8c-kube-api-access-s9bd8\") pod \"iptables-alerter-tcn8w\" (UID: \"a734dd47-fffc-4dfd-8946-3c3466565b8c\") " pod="openshift-network-operator/iptables-alerter-tcn8w"
Jul 31 14:48:26.069114 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.068838    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4c2a7673-4169-4edb-a234-70c7d12e0804-registration-dir\") pod \"aws-ebs-csi-driver-node-wmw57\" (UID: \"4c2a7673-4169-4edb-a234-70c7d12e0804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57"
Jul 31 14:48:26.069114 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.068863    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54adc061-d1ce-4ba7-83cf-3b2968693d52-tmp-dir\") pod \"node-resolver-mpkxl\" (UID: \"54adc061-d1ce-4ba7-83cf-3b2968693d52\") " pod="openshift-dns/node-resolver-mpkxl"
Jul 31 14:48:26.070579 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.070556    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 31 14:48:26.071021 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.071001    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 31 14:48:26.071408 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.071391    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.072674 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.072654    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:26.072840 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:26.072728    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk8pw" podUID="d947335a-f9d7-4fcd-b9e1-457477ab7f72"
Jul 31 14:48:26.074712 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.074695    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lvzsk\""
Jul 31 14:48:26.075202 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.075185    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.077714 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.077696    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jfrf7"
Jul 31 14:48:26.080261 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.080232    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.085202 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.085179    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 31 14:48:26.085202 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.085196    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 31 14:48:26.085395 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.085375    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 31 14:48:26.085479 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.085438    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 31 14:48:26.085532 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.085519    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 31 14:48:26.085600 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.085582    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 31 14:48:26.085650 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.085385    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pkb8m\""
Jul 31 14:48:26.085703 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.085672    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 31 14:48:26.085754 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.085704    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 31 14:48:26.086012 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.085988    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ctwcr\""
Jul 31 14:48:26.086127 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.086099    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 31 14:48:26.086225 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.086106    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 31 14:48:26.086412 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.086395    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9hzt8\""
Jul 31 14:48:26.086575 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.086562    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 31 14:48:26.086775 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.086759    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 31 14:48:26.086852 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.086837    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:48:26.087046 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.087029    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 31 14:48:26.087178 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.087159    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wv8q6\""
Jul 31 14:48:26.108548 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.108526    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 14:48:26.163617 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.163584    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 31 14:48:26.169044 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169008    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-var-lib-openvswitch\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.169174 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169052    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-etc-openvswitch\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.169174 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169080    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d947335a-f9d7-4fcd-b9e1-457477ab7f72-dbus\") pod \"global-pull-secret-syncer-nk8pw\" (UID: \"d947335a-f9d7-4fcd-b9e1-457477ab7f72\") " pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:26.169174 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169107    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-var-lib-kubelet\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.169174 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169140    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4c2a7673-4169-4edb-a234-70c7d12e0804-sys-fs\") pod \"aws-ebs-csi-driver-node-wmw57\" (UID: \"4c2a7673-4169-4edb-a234-70c7d12e0804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57"
Jul 31 14:48:26.169174 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169166    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-host-run-netns\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.169395 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169190    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-host-cni-netd\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.169395 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169215    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/38a6d9ec-ffc7-4d77-b8c2-819775c1cf24-cnibin\") pod \"multus-additional-cni-plugins-jfrf7\" (UID: \"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24\") " pod="openshift-multus/multus-additional-cni-plugins-jfrf7"
Jul 31 14:48:26.169395 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169241    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhqsp\" (UniqueName: \"kubernetes.io/projected/16b5669e-0dde-49c0-854b-495854e99a82-kube-api-access-dhqsp\") pod \"node-ca-bwcjx\" (UID: \"16b5669e-0dde-49c0-854b-495854e99a82\") " pod="openshift-image-registry/node-ca-bwcjx"
Jul 31 14:48:26.169395 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169265    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-log-socket\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.169395 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169299    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-host-run-k8s-cni-cncf-io\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.169395 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169326    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4c2a7673-4169-4edb-a234-70c7d12e0804-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wmw57\" (UID: \"4c2a7673-4169-4edb-a234-70c7d12e0804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57"
Jul 31 14:48:26.169395 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169352    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqpwq\" (UniqueName: \"kubernetes.io/projected/54adc061-d1ce-4ba7-83cf-3b2968693d52-kube-api-access-sqpwq\") pod \"node-resolver-mpkxl\" (UID: \"54adc061-d1ce-4ba7-83cf-3b2968693d52\") " pod="openshift-dns/node-resolver-mpkxl"
Jul 31 14:48:26.169395 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169376    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-run-systemd\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.169723 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169399    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-etc-kubernetes\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.169723 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169421    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-sys\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.169723 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169446    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f6546deb-c60b-4bbb-9535-7dac6a479299-etc-tuned\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.169723 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169468    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6546deb-c60b-4bbb-9535-7dac6a479299-tmp\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.169723 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169490    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-multus-cni-dir\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.169723 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169517    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-etc-sysctl-conf\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.169723 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169540    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-run\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.169723 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169561    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-host\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.169723 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169585    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-host-var-lib-cni-bin\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.169723 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169611    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp2dl\" (UniqueName: \"kubernetes.io/projected/38a6d9ec-ffc7-4d77-b8c2-819775c1cf24-kube-api-access-gp2dl\") pod \"multus-additional-cni-plugins-jfrf7\" (UID: \"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24\") " pod="openshift-multus/multus-additional-cni-plugins-jfrf7"
Jul 31 14:48:26.169723 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169638    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9bd8\" (UniqueName: \"kubernetes.io/projected/a734dd47-fffc-4dfd-8946-3c3466565b8c-kube-api-access-s9bd8\") pod \"iptables-alerter-tcn8w\" (UID: \"a734dd47-fffc-4dfd-8946-3c3466565b8c\") " pod="openshift-network-operator/iptables-alerter-tcn8w"
Jul 31 14:48:26.169723 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169664    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/38a6d9ec-ffc7-4d77-b8c2-819775c1cf24-cni-binary-copy\") pod \"multus-additional-cni-plugins-jfrf7\" (UID: \"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24\") " pod="openshift-multus/multus-additional-cni-plugins-jfrf7"
Jul 31 14:48:26.169723 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169689    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-etc-modprobe-d\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.169723 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169715    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/988269ba-5182-484e-ad71-c72e50c2fe3b-agent-certs\") pod \"konnectivity-agent-xg6w7\" (UID: \"988269ba-5182-484e-ad71-c72e50c2fe3b\") " pod="kube-system/konnectivity-agent-xg6w7"
Jul 31 14:48:26.170417 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169741    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4c2a7673-4169-4edb-a234-70c7d12e0804-device-dir\") pod \"aws-ebs-csi-driver-node-wmw57\" (UID: \"4c2a7673-4169-4edb-a234-70c7d12e0804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57"
Jul 31 14:48:26.170417 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169772    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16b5669e-0dde-49c0-854b-495854e99a82-host\") pod \"node-ca-bwcjx\" (UID: \"16b5669e-0dde-49c0-854b-495854e99a82\") " pod="openshift-image-registry/node-ca-bwcjx"
Jul 31 14:48:26.170417 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169825    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/38a6d9ec-ffc7-4d77-b8c2-819775c1cf24-os-release\") pod \"multus-additional-cni-plugins-jfrf7\" (UID: \"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24\") " pod="openshift-multus/multus-additional-cni-plugins-jfrf7"
Jul 31 14:48:26.170417 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169941    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4c2a7673-4169-4edb-a234-70c7d12e0804-device-dir\") pod \"aws-ebs-csi-driver-node-wmw57\" (UID: \"4c2a7673-4169-4edb-a234-70c7d12e0804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57"
Jul 31 14:48:26.170417 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.169963    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16b5669e-0dde-49c0-854b-495854e99a82-host\") pod \"node-ca-bwcjx\" (UID: \"16b5669e-0dde-49c0-854b-495854e99a82\") " pod="openshift-image-registry/node-ca-bwcjx"
Jul 31 14:48:26.170417 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.170000    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4c2a7673-4169-4edb-a234-70c7d12e0804-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wmw57\" (UID: \"4c2a7673-4169-4edb-a234-70c7d12e0804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57"
Jul 31 14:48:26.170417 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.170037    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/38a6d9ec-ffc7-4d77-b8c2-819775c1cf24-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jfrf7\" (UID: \"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24\") " pod="openshift-multus/multus-additional-cni-plugins-jfrf7"
Jul 31 14:48:26.170417 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.170105    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-etc-sysconfig\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.170417 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.170150    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97xf4\" (UniqueName: \"kubernetes.io/projected/567a8972-6aaf-499f-a8d8-a83da117da60-kube-api-access-97xf4\") pod \"network-check-target-6vfsr\" (UID: \"567a8972-6aaf-499f-a8d8-a83da117da60\") " pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:48:26.170417 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.170183    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4c2a7673-4169-4edb-a234-70c7d12e0804-sys-fs\") pod \"aws-ebs-csi-driver-node-wmw57\" (UID: \"4c2a7673-4169-4edb-a234-70c7d12e0804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57"
Jul 31 14:48:26.170417 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.170238    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-host-run-netns\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.170417 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.170299    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4c2a7673-4169-4edb-a234-70c7d12e0804-socket-dir\") pod \"aws-ebs-csi-driver-node-wmw57\" (UID: \"4c2a7673-4169-4edb-a234-70c7d12e0804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57"
Jul 31 14:48:26.170417 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.170343    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zshq2\" (UniqueName: \"kubernetes.io/projected/355d9cbb-1867-49cc-8ec2-564b4aa6daea-kube-api-access-zshq2\") pod \"network-metrics-daemon-b45r4\" (UID: \"355d9cbb-1867-49cc-8ec2-564b4aa6daea\") " pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:26.170417 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.170375    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c55da51d-eba3-4993-a0bb-65f32e5d73cc-ovnkube-config\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.170417 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.170425    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c55da51d-eba3-4993-a0bb-65f32e5d73cc-env-overrides\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.171184 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.170453    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d947335a-f9d7-4fcd-b9e1-457477ab7f72-kubelet-config\") pod \"global-pull-secret-syncer-nk8pw\" (UID: \"d947335a-f9d7-4fcd-b9e1-457477ab7f72\") " pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:26.171184 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.170474    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-cnibin\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.171184 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.170486    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4c2a7673-4169-4edb-a234-70c7d12e0804-socket-dir\") pod \"aws-ebs-csi-driver-node-wmw57\" (UID: \"4c2a7673-4169-4edb-a234-70c7d12e0804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57"
Jul 31 14:48:26.171184 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.170514    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-host-var-lib-kubelet\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.171184 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.170543    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4c2a7673-4169-4edb-a234-70c7d12e0804-etc-selinux\") pod \"aws-ebs-csi-driver-node-wmw57\" (UID: \"4c2a7673-4169-4edb-a234-70c7d12e0804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57"
Jul 31 14:48:26.171184 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.170571    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb7ng\" (UniqueName: \"kubernetes.io/projected/4c2a7673-4169-4edb-a234-70c7d12e0804-kube-api-access-nb7ng\") pod \"aws-ebs-csi-driver-node-wmw57\" (UID: \"4c2a7673-4169-4edb-a234-70c7d12e0804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57"
Jul 31 14:48:26.171184 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.170599    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.171184 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.170628    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-host-var-lib-cni-multus\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.171184 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.170675    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54adc061-d1ce-4ba7-83cf-3b2968693d52-tmp-dir\") pod \"node-resolver-mpkxl\" (UID: \"54adc061-d1ce-4ba7-83cf-3b2968693d52\") " pod="openshift-dns/node-resolver-mpkxl"
Jul 31 14:48:26.171184 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.170702    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-systemd-units\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.171184 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.170734    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-474jn\" (UniqueName: \"kubernetes.io/projected/c55da51d-eba3-4993-a0bb-65f32e5d73cc-kube-api-access-474jn\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.171184 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.170785    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/38a6d9ec-ffc7-4d77-b8c2-819775c1cf24-system-cni-dir\") pod \"multus-additional-cni-plugins-jfrf7\" (UID: \"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24\") " pod="openshift-multus/multus-additional-cni-plugins-jfrf7"
Jul 31 14:48:26.171184 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.170832    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/38a6d9ec-ffc7-4d77-b8c2-819775c1cf24-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jfrf7\" (UID: \"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24\") " pod="openshift-multus/multus-additional-cni-plugins-jfrf7"
Jul 31 14:48:26.171184 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.170866    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a734dd47-fffc-4dfd-8946-3c3466565b8c-host-slash\") pod \"iptables-alerter-tcn8w\" (UID: \"a734dd47-fffc-4dfd-8946-3c3466565b8c\") " pod="openshift-network-operator/iptables-alerter-tcn8w"
Jul 31 14:48:26.171184 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.170903    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a734dd47-fffc-4dfd-8946-3c3466565b8c-host-slash\") pod \"iptables-alerter-tcn8w\" (UID: \"a734dd47-fffc-4dfd-8946-3c3466565b8c\") " pod="openshift-network-operator/iptables-alerter-tcn8w"
Jul 31 14:48:26.171184 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.170965    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54adc061-d1ce-4ba7-83cf-3b2968693d52-tmp-dir\") pod \"node-resolver-mpkxl\" (UID: \"54adc061-d1ce-4ba7-83cf-3b2968693d52\") " pod="openshift-dns/node-resolver-mpkxl"
Jul 31 14:48:26.171184 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171007    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4c2a7673-4169-4edb-a234-70c7d12e0804-etc-selinux\") pod \"aws-ebs-csi-driver-node-wmw57\" (UID: \"4c2a7673-4169-4edb-a234-70c7d12e0804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57"
Jul 31 14:48:26.172054 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171068    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c55da51d-eba3-4993-a0bb-65f32e5d73cc-ovn-node-metrics-cert\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.172054 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171099    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c55da51d-eba3-4993-a0bb-65f32e5d73cc-ovnkube-script-lib\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.172054 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171131    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-os-release\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.172054 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171176    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-multus-socket-dir-parent\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.172054 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171204    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv8hj\" (UniqueName: \"kubernetes.io/projected/88ce38fc-0695-4bfe-8292-3547c26c44d5-kube-api-access-wv8hj\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.172054 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171232    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a734dd47-fffc-4dfd-8946-3c3466565b8c-iptables-alerter-script\") pod \"iptables-alerter-tcn8w\" (UID: \"a734dd47-fffc-4dfd-8946-3c3466565b8c\") " pod="openshift-network-operator/iptables-alerter-tcn8w"
Jul 31 14:48:26.172054 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171256    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-host-kubelet\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.172054 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171278    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-run-ovn\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.172054 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171309    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d947335a-f9d7-4fcd-b9e1-457477ab7f72-original-pull-secret\") pod \"global-pull-secret-syncer-nk8pw\" (UID: \"d947335a-f9d7-4fcd-b9e1-457477ab7f72\") " pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:26.172054 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171331    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/38a6d9ec-ffc7-4d77-b8c2-819775c1cf24-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jfrf7\" (UID: \"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24\") " pod="openshift-multus/multus-additional-cni-plugins-jfrf7"
Jul 31 14:48:26.172054 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171359    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/54adc061-d1ce-4ba7-83cf-3b2968693d52-hosts-file\") pod \"node-resolver-mpkxl\" (UID: \"54adc061-d1ce-4ba7-83cf-3b2968693d52\") " pod="openshift-dns/node-resolver-mpkxl"
Jul 31 14:48:26.172054 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171383    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-host-slash\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.172054 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171406    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-run-openvswitch\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.172054 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171431    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-node-log\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.172054 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171465    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-etc-sysctl-d\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.172054 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171482    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bllc\" (UniqueName: \"kubernetes.io/projected/f6546deb-c60b-4bbb-9535-7dac6a479299-kube-api-access-8bllc\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.172750 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171498    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-host-run-multus-certs\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.172750 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171514    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-etc-kubernetes\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.172750 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171533    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4c2a7673-4169-4edb-a234-70c7d12e0804-registration-dir\") pod \"aws-ebs-csi-driver-node-wmw57\" (UID: \"4c2a7673-4169-4edb-a234-70c7d12e0804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57"
Jul 31 14:48:26.172750 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171557    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-host-run-ovn-kubernetes\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.172750 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171605    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-host-cni-bin\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.172750 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171624    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-etc-systemd\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.172750 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171640    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-lib-modules\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.172750 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171674    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/988269ba-5182-484e-ad71-c72e50c2fe3b-konnectivity-ca\") pod \"konnectivity-agent-xg6w7\" (UID: \"988269ba-5182-484e-ad71-c72e50c2fe3b\") " pod="kube-system/konnectivity-agent-xg6w7"
Jul 31 14:48:26.172750 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171700    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-system-cni-dir\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.172750 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171725    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88ce38fc-0695-4bfe-8292-3547c26c44d5-cni-binary-copy\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.172750 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171750    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-hostroot\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.172750 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171776    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-multus-conf-dir\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.172750 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171797    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/88ce38fc-0695-4bfe-8292-3547c26c44d5-multus-daemon-config\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.172750 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171834    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/16b5669e-0dde-49c0-854b-495854e99a82-serviceca\") pod \"node-ca-bwcjx\" (UID: \"16b5669e-0dde-49c0-854b-495854e99a82\") " pod="openshift-image-registry/node-ca-bwcjx"
Jul 31 14:48:26.172750 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171852    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355d9cbb-1867-49cc-8ec2-564b4aa6daea-metrics-certs\") pod \"network-metrics-daemon-b45r4\" (UID: \"355d9cbb-1867-49cc-8ec2-564b4aa6daea\") " pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:26.172750 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171890    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/54adc061-d1ce-4ba7-83cf-3b2968693d52-hosts-file\") pod \"node-resolver-mpkxl\" (UID: \"54adc061-d1ce-4ba7-83cf-3b2968693d52\") " pod="openshift-dns/node-resolver-mpkxl"
Jul 31 14:48:26.172750 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:26.171934    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:26.173427 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.171966    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4c2a7673-4169-4edb-a234-70c7d12e0804-registration-dir\") pod \"aws-ebs-csi-driver-node-wmw57\" (UID: \"4c2a7673-4169-4edb-a234-70c7d12e0804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57"
Jul 31 14:48:26.173427 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:26.171985    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/355d9cbb-1867-49cc-8ec2-564b4aa6daea-metrics-certs podName:355d9cbb-1867-49cc-8ec2-564b4aa6daea nodeName:}" failed. No retries permitted until 2026-07-31 14:48:26.671969312 +0000 UTC m=+2.983526914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/355d9cbb-1867-49cc-8ec2-564b4aa6daea-metrics-certs") pod "network-metrics-daemon-b45r4" (UID: "355d9cbb-1867-49cc-8ec2-564b4aa6daea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:26.173427 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.172086    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a734dd47-fffc-4dfd-8946-3c3466565b8c-iptables-alerter-script\") pod \"iptables-alerter-tcn8w\" (UID: \"a734dd47-fffc-4dfd-8946-3c3466565b8c\") " pod="openshift-network-operator/iptables-alerter-tcn8w"
Jul 31 14:48:26.173427 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.172488    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/16b5669e-0dde-49c0-854b-495854e99a82-serviceca\") pod \"node-ca-bwcjx\" (UID: \"16b5669e-0dde-49c0-854b-495854e99a82\") " pod="openshift-image-registry/node-ca-bwcjx"
Jul 31 14:48:26.185868 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.185844    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6xzs5"
Jul 31 14:48:26.204862 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.204838    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6xzs5"
Jul 31 14:48:26.205287 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.205255    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 31 14:48:26.209262 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.209241    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zshq2\" (UniqueName: \"kubernetes.io/projected/355d9cbb-1867-49cc-8ec2-564b4aa6daea-kube-api-access-zshq2\") pod \"network-metrics-daemon-b45r4\" (UID: \"355d9cbb-1867-49cc-8ec2-564b4aa6daea\") " pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:26.210150 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.210125    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9bd8\" (UniqueName: \"kubernetes.io/projected/a734dd47-fffc-4dfd-8946-3c3466565b8c-kube-api-access-s9bd8\") pod \"iptables-alerter-tcn8w\" (UID: \"a734dd47-fffc-4dfd-8946-3c3466565b8c\") " pod="openshift-network-operator/iptables-alerter-tcn8w"
Jul 31 14:48:26.210421 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.210400    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb7ng\" (UniqueName: \"kubernetes.io/projected/4c2a7673-4169-4edb-a234-70c7d12e0804-kube-api-access-nb7ng\") pod \"aws-ebs-csi-driver-node-wmw57\" (UID: \"4c2a7673-4169-4edb-a234-70c7d12e0804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57"
Jul 31 14:48:26.211901 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.211869    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhqsp\" (UniqueName: \"kubernetes.io/projected/16b5669e-0dde-49c0-854b-495854e99a82-kube-api-access-dhqsp\") pod \"node-ca-bwcjx\" (UID: \"16b5669e-0dde-49c0-854b-495854e99a82\") " pod="openshift-image-registry/node-ca-bwcjx"
Jul 31 14:48:26.212346 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.212326    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqpwq\" (UniqueName: \"kubernetes.io/projected/54adc061-d1ce-4ba7-83cf-3b2968693d52-kube-api-access-sqpwq\") pod \"node-resolver-mpkxl\" (UID: \"54adc061-d1ce-4ba7-83cf-3b2968693d52\") " pod="openshift-dns/node-resolver-mpkxl"
Jul 31 14:48:26.272277 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272238    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-log-socket\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.272471 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272286    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-host-run-k8s-cni-cncf-io\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.272471 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272315    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-run-systemd\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.272471 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272336    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-etc-kubernetes\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.272471 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272359    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-sys\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.272471 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272332    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-log-socket\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.272471 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272382    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f6546deb-c60b-4bbb-9535-7dac6a479299-etc-tuned\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.272471 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272424    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6546deb-c60b-4bbb-9535-7dac6a479299-tmp\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.272471 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272441    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-host-run-k8s-cni-cncf-io\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.272471 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272455    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-etc-kubernetes\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.272471 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272460    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-sys\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.272471 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272472    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-multus-cni-dir\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.273070 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272516    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-etc-sysctl-conf\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.273070 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272542    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-run\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.273070 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272458    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-run-systemd\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.273070 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272582    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-host\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.273070 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272605    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-host-var-lib-cni-bin\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.273070 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272628    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp2dl\" (UniqueName: \"kubernetes.io/projected/38a6d9ec-ffc7-4d77-b8c2-819775c1cf24-kube-api-access-gp2dl\") pod \"multus-additional-cni-plugins-jfrf7\" (UID: \"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24\") " pod="openshift-multus/multus-additional-cni-plugins-jfrf7"
Jul 31 14:48:26.273070 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272656    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/38a6d9ec-ffc7-4d77-b8c2-819775c1cf24-cni-binary-copy\") pod \"multus-additional-cni-plugins-jfrf7\" (UID: \"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24\") " pod="openshift-multus/multus-additional-cni-plugins-jfrf7"
Jul 31 14:48:26.273070 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272672    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-host\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.273070 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272685    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-etc-modprobe-d\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.273070 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272697    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-etc-sysctl-conf\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.273070 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272710    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/988269ba-5182-484e-ad71-c72e50c2fe3b-agent-certs\") pod \"konnectivity-agent-xg6w7\" (UID: \"988269ba-5182-484e-ad71-c72e50c2fe3b\") " pod="kube-system/konnectivity-agent-xg6w7"
Jul 31 14:48:26.273070 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272724    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-run\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.273070 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272741    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/38a6d9ec-ffc7-4d77-b8c2-819775c1cf24-os-release\") pod \"multus-additional-cni-plugins-jfrf7\" (UID: \"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24\") " pod="openshift-multus/multus-additional-cni-plugins-jfrf7"
Jul 31 14:48:26.273070 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272757    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-host-var-lib-cni-bin\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.273070 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272793    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/38a6d9ec-ffc7-4d77-b8c2-819775c1cf24-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jfrf7\" (UID: \"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24\") " pod="openshift-multus/multus-additional-cni-plugins-jfrf7"
Jul 31 14:48:26.273070 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272820    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-etc-sysconfig\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.273070 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272853    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97xf4\" (UniqueName: \"kubernetes.io/projected/567a8972-6aaf-499f-a8d8-a83da117da60-kube-api-access-97xf4\") pod \"network-check-target-6vfsr\" (UID: \"567a8972-6aaf-499f-a8d8-a83da117da60\") " pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:48:26.274218 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272861    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-etc-modprobe-d\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.274218 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272878    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-host-run-netns\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.274218 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272908    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c55da51d-eba3-4993-a0bb-65f32e5d73cc-ovnkube-config\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.274218 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272933    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/38a6d9ec-ffc7-4d77-b8c2-819775c1cf24-os-release\") pod \"multus-additional-cni-plugins-jfrf7\" (UID: \"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24\") " pod="openshift-multus/multus-additional-cni-plugins-jfrf7"
Jul 31 14:48:26.274218 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272934    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c55da51d-eba3-4993-a0bb-65f32e5d73cc-env-overrides\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.274218 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272979    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d947335a-f9d7-4fcd-b9e1-457477ab7f72-kubelet-config\") pod \"global-pull-secret-syncer-nk8pw\" (UID: \"d947335a-f9d7-4fcd-b9e1-457477ab7f72\") " pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:26.274218 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273006    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-cnibin\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.274218 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273032    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-host-var-lib-kubelet\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.274218 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273062    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.274218 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273088    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-host-var-lib-cni-multus\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.274218 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273115    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-systemd-units\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.274218 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273143    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-474jn\" (UniqueName: \"kubernetes.io/projected/c55da51d-eba3-4993-a0bb-65f32e5d73cc-kube-api-access-474jn\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.274218 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273172    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/38a6d9ec-ffc7-4d77-b8c2-819775c1cf24-system-cni-dir\") pod \"multus-additional-cni-plugins-jfrf7\" (UID: \"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24\") " pod="openshift-multus/multus-additional-cni-plugins-jfrf7"
Jul 31 14:48:26.274218 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273198    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/38a6d9ec-ffc7-4d77-b8c2-819775c1cf24-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jfrf7\" (UID: \"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24\") " pod="openshift-multus/multus-additional-cni-plugins-jfrf7"
Jul 31 14:48:26.274218 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273257    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c55da51d-eba3-4993-a0bb-65f32e5d73cc-ovn-node-metrics-cert\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.274218 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273376    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c55da51d-eba3-4993-a0bb-65f32e5d73cc-env-overrides\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.274218 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273391    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.275055 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273402    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/38a6d9ec-ffc7-4d77-b8c2-819775c1cf24-cni-binary-copy\") pod \"multus-additional-cni-plugins-jfrf7\" (UID: \"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24\") " pod="openshift-multus/multus-additional-cni-plugins-jfrf7"
Jul 31 14:48:26.275055 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.272628    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-multus-cni-dir\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.275055 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273395    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c55da51d-eba3-4993-a0bb-65f32e5d73cc-ovnkube-script-lib\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.275055 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273451    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-os-release\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.275055 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273462    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-host-var-lib-kubelet\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.275055 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273463    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-etc-sysconfig\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.275055 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273475    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-multus-socket-dir-parent\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.275055 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273509    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-cnibin\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.275055 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273520    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv8hj\" (UniqueName: \"kubernetes.io/projected/88ce38fc-0695-4bfe-8292-3547c26c44d5-kube-api-access-wv8hj\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.275055 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273527    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-multus-socket-dir-parent\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.275055 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273552    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-host-kubelet\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.275055 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273576    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-host-run-netns\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.275055 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273581    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-run-ovn\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.275055 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273651    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-systemd-units\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.275055 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273727    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d947335a-f9d7-4fcd-b9e1-457477ab7f72-original-pull-secret\") pod \"global-pull-secret-syncer-nk8pw\" (UID: \"d947335a-f9d7-4fcd-b9e1-457477ab7f72\") " pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:26.275055 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273829    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/38a6d9ec-ffc7-4d77-b8c2-819775c1cf24-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jfrf7\" (UID: \"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24\") " pod="openshift-multus/multus-additional-cni-plugins-jfrf7"
Jul 31 14:48:26.275055 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273860    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-host-slash\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.275055 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273868    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/38a6d9ec-ffc7-4d77-b8c2-819775c1cf24-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jfrf7\" (UID: \"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24\") " pod="openshift-multus/multus-additional-cni-plugins-jfrf7"
Jul 31 14:48:26.276036 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273886    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-run-openvswitch\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.276036 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273918    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-node-log\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.276036 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273936    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d947335a-f9d7-4fcd-b9e1-457477ab7f72-kubelet-config\") pod \"global-pull-secret-syncer-nk8pw\" (UID: \"d947335a-f9d7-4fcd-b9e1-457477ab7f72\") " pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:26.276036 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273944    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-etc-sysctl-d\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.276036 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273972    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bllc\" (UniqueName: \"kubernetes.io/projected/f6546deb-c60b-4bbb-9535-7dac6a479299-kube-api-access-8bllc\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.276036 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274000    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-host-run-multus-certs\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.276036 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274028    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-etc-kubernetes\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.276036 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274058    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-host-run-ovn-kubernetes\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.276036 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274092    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c55da51d-eba3-4993-a0bb-65f32e5d73cc-ovnkube-config\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.276036 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274108    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-host-cni-bin\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.276036 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274144    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-etc-systemd\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.276036 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274156    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-node-log\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.276036 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274165    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-os-release\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.276036 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274172    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-lib-modules\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.276036 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274327    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-etc-sysctl-d\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.276036 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274333    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/988269ba-5182-484e-ad71-c72e50c2fe3b-konnectivity-ca\") pod \"konnectivity-agent-xg6w7\" (UID: \"988269ba-5182-484e-ad71-c72e50c2fe3b\") " pod="kube-system/konnectivity-agent-xg6w7"
Jul 31 14:48:26.276036 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274110    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/38a6d9ec-ffc7-4d77-b8c2-819775c1cf24-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jfrf7\" (UID: \"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24\") " pod="openshift-multus/multus-additional-cni-plugins-jfrf7"
Jul 31 14:48:26.276036 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274372    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-system-cni-dir\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.277001 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274400    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88ce38fc-0695-4bfe-8292-3547c26c44d5-cni-binary-copy\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.277001 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274417    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-etc-systemd\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.277001 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274457    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-host-kubelet\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.277001 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274495    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-hostroot\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.277001 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274527    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-multus-conf-dir\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.277001 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274555    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/88ce38fc-0695-4bfe-8292-3547c26c44d5-multus-daemon-config\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.277001 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274600    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-var-lib-openvswitch\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.277001 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274630    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-etc-openvswitch\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.277001 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274669    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d947335a-f9d7-4fcd-b9e1-457477ab7f72-dbus\") pod \"global-pull-secret-syncer-nk8pw\" (UID: \"d947335a-f9d7-4fcd-b9e1-457477ab7f72\") " pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:26.277001 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274698    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-var-lib-kubelet\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.277001 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274711    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-system-cni-dir\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.277001 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274334    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-etc-kubernetes\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.277001 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274736    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-host-run-netns\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.277001 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274763    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-host-cni-netd\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.277001 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274775    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-host-run-ovn-kubernetes\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.277001 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274789    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/38a6d9ec-ffc7-4d77-b8c2-819775c1cf24-cnibin\") pod \"multus-additional-cni-plugins-jfrf7\" (UID: \"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24\") " pod="openshift-multus/multus-additional-cni-plugins-jfrf7"
Jul 31 14:48:26.277001 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274837    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-hostroot\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.277001 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274878    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-run-ovn\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.277969 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274884    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c55da51d-eba3-4993-a0bb-65f32e5d73cc-ovnkube-script-lib\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.277969 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273618    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-host-var-lib-cni-multus\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.277969 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274891    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/38a6d9ec-ffc7-4d77-b8c2-819775c1cf24-cnibin\") pod \"multus-additional-cni-plugins-jfrf7\" (UID: \"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24\") " pod="openshift-multus/multus-additional-cni-plugins-jfrf7"
Jul 31 14:48:26.277969 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274930    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-var-lib-openvswitch\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.277969 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274946    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-host-run-multus-certs\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.277969 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:26.274971    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:26.277969 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.274981    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-etc-openvswitch\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.277969 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.275049    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d947335a-f9d7-4fcd-b9e1-457477ab7f72-dbus\") pod \"global-pull-secret-syncer-nk8pw\" (UID: \"d947335a-f9d7-4fcd-b9e1-457477ab7f72\") " pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:26.277969 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.275100    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-var-lib-kubelet\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.277969 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:26.275111    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d947335a-f9d7-4fcd-b9e1-457477ab7f72-original-pull-secret podName:d947335a-f9d7-4fcd-b9e1-457477ab7f72 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:26.775072974 +0000 UTC m=+3.086630580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d947335a-f9d7-4fcd-b9e1-457477ab7f72-original-pull-secret") pod "global-pull-secret-syncer-nk8pw" (UID: "d947335a-f9d7-4fcd-b9e1-457477ab7f72") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:26.277969 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.275147    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-host-run-netns\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.277969 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.275147    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/988269ba-5182-484e-ad71-c72e50c2fe3b-konnectivity-ca\") pod \"konnectivity-agent-xg6w7\" (UID: \"988269ba-5182-484e-ad71-c72e50c2fe3b\") " pod="kube-system/konnectivity-agent-xg6w7"
Jul 31 14:48:26.277969 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.273943    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/38a6d9ec-ffc7-4d77-b8c2-819775c1cf24-system-cni-dir\") pod \"multus-additional-cni-plugins-jfrf7\" (UID: \"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24\") " pod="openshift-multus/multus-additional-cni-plugins-jfrf7"
Jul 31 14:48:26.277969 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.275209    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-host-cni-netd\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.277969 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.275264    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-run-openvswitch\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.277969 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.275343    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f6546deb-c60b-4bbb-9535-7dac6a479299-lib-modules\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.277969 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.275400    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-host-slash\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.278855 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.275438    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/88ce38fc-0695-4bfe-8292-3547c26c44d5-multus-conf-dir\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.278855 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.275441    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f6546deb-c60b-4bbb-9535-7dac6a479299-etc-tuned\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.278855 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.275525    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/88ce38fc-0695-4bfe-8292-3547c26c44d5-multus-daemon-config\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.278855 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.275577    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c55da51d-eba3-4993-a0bb-65f32e5d73cc-host-cni-bin\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.278855 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.275659    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/88ce38fc-0695-4bfe-8292-3547c26c44d5-cni-binary-copy\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.278855 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.276508    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6546deb-c60b-4bbb-9535-7dac6a479299-tmp\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.278855 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.276559    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/988269ba-5182-484e-ad71-c72e50c2fe3b-agent-certs\") pod \"konnectivity-agent-xg6w7\" (UID: \"988269ba-5182-484e-ad71-c72e50c2fe3b\") " pod="kube-system/konnectivity-agent-xg6w7"
Jul 31 14:48:26.278855 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.276792    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/38a6d9ec-ffc7-4d77-b8c2-819775c1cf24-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jfrf7\" (UID: \"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24\") " pod="openshift-multus/multus-additional-cni-plugins-jfrf7"
Jul 31 14:48:26.278855 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.278025    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c55da51d-eba3-4993-a0bb-65f32e5d73cc-ovn-node-metrics-cert\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.282765 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.282740    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp2dl\" (UniqueName: \"kubernetes.io/projected/38a6d9ec-ffc7-4d77-b8c2-819775c1cf24-kube-api-access-gp2dl\") pod \"multus-additional-cni-plugins-jfrf7\" (UID: \"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24\") " pod="openshift-multus/multus-additional-cni-plugins-jfrf7"
Jul 31 14:48:26.283118 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.283049    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-474jn\" (UniqueName: \"kubernetes.io/projected/c55da51d-eba3-4993-a0bb-65f32e5d73cc-kube-api-access-474jn\") pod \"ovnkube-node-qp8cd\" (UID: \"c55da51d-eba3-4993-a0bb-65f32e5d73cc\") " pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.283308 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:26.283285    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:26.283396 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:26.283312    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:26.283396 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:26.283326    2559 projected.go:196] Error preparing data for projected volume kube-api-access-97xf4 for pod openshift-network-diagnostics/network-check-target-6vfsr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:26.283396 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:26.283389    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/567a8972-6aaf-499f-a8d8-a83da117da60-kube-api-access-97xf4 podName:567a8972-6aaf-499f-a8d8-a83da117da60 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:26.783375573 +0000 UTC m=+3.094933175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-97xf4" (UniqueName: "kubernetes.io/projected/567a8972-6aaf-499f-a8d8-a83da117da60-kube-api-access-97xf4") pod "network-check-target-6vfsr" (UID: "567a8972-6aaf-499f-a8d8-a83da117da60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:26.285553 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.285532    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv8hj\" (UniqueName: \"kubernetes.io/projected/88ce38fc-0695-4bfe-8292-3547c26c44d5-kube-api-access-wv8hj\") pod \"multus-64wqp\" (UID: \"88ce38fc-0695-4bfe-8292-3547c26c44d5\") " pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.286210 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.286182    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bllc\" (UniqueName: \"kubernetes.io/projected/f6546deb-c60b-4bbb-9535-7dac6a479299-kube-api-access-8bllc\") pod \"tuned-xvpvq\" (UID: \"f6546deb-c60b-4bbb-9535-7dac6a479299\") " pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.362345 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.362265    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tcn8w"
Jul 31 14:48:26.375065 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.375027    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57"
Jul 31 14:48:26.382369 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.382344    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mpkxl"
Jul 31 14:48:26.387673 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.387650    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bwcjx"
Jul 31 14:48:26.401695 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.401667    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xg6w7"
Jul 31 14:48:26.407938 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.407918    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-64wqp"
Jul 31 14:48:26.417050 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.417027    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:26.424297 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.424274    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jfrf7"
Jul 31 14:48:26.429152 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.429128    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xvpvq"
Jul 31 14:48:26.678317 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.678221    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355d9cbb-1867-49cc-8ec2-564b4aa6daea-metrics-certs\") pod \"network-metrics-daemon-b45r4\" (UID: \"355d9cbb-1867-49cc-8ec2-564b4aa6daea\") " pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:26.678500 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:26.678372    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:26.678500 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:26.678443    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/355d9cbb-1867-49cc-8ec2-564b4aa6daea-metrics-certs podName:355d9cbb-1867-49cc-8ec2-564b4aa6daea nodeName:}" failed. No retries permitted until 2026-07-31 14:48:27.678427132 +0000 UTC m=+3.989984740 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/355d9cbb-1867-49cc-8ec2-564b4aa6daea-metrics-certs") pod "network-metrics-daemon-b45r4" (UID: "355d9cbb-1867-49cc-8ec2-564b4aa6daea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:26.763969 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:26.763933    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6546deb_c60b_4bbb_9535_7dac6a479299.slice/crio-609e7259c447776087ba4ed17d508c1e98f7ca0137a31bf596d4bfc48d1c9f82 WatchSource:0}: Error finding container 609e7259c447776087ba4ed17d508c1e98f7ca0137a31bf596d4bfc48d1c9f82: Status 404 returned error can't find the container with id 609e7259c447776087ba4ed17d508c1e98f7ca0137a31bf596d4bfc48d1c9f82
Jul 31 14:48:26.765645 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:26.765622    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a6d9ec_ffc7_4d77_b8c2_819775c1cf24.slice/crio-c50c1b896513d703775de91e82d53d5c446d11bad736bb1d9c6b90375cbe6ad4 WatchSource:0}: Error finding container c50c1b896513d703775de91e82d53d5c446d11bad736bb1d9c6b90375cbe6ad4: Status 404 returned error can't find the container with id c50c1b896513d703775de91e82d53d5c446d11bad736bb1d9c6b90375cbe6ad4
Jul 31 14:48:26.766468 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:26.766435    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b5669e_0dde_49c0_854b_495854e99a82.slice/crio-4bec6b0cbd24ca34afd60d41f949af8c370f416117429aa361dde0322ed805c3 WatchSource:0}: Error finding container 4bec6b0cbd24ca34afd60d41f949af8c370f416117429aa361dde0322ed805c3: Status 404 returned error can't find the container with id 4bec6b0cbd24ca34afd60d41f949af8c370f416117429aa361dde0322ed805c3
Jul 31 14:48:26.768916 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:26.768892    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc55da51d_eba3_4993_a0bb_65f32e5d73cc.slice/crio-a48dacd1bda2284e4c0bd65cd22996f41a7ae6e49a2c9cc6728df467209184a6 WatchSource:0}: Error finding container a48dacd1bda2284e4c0bd65cd22996f41a7ae6e49a2c9cc6728df467209184a6: Status 404 returned error can't find the container with id a48dacd1bda2284e4c0bd65cd22996f41a7ae6e49a2c9cc6728df467209184a6
Jul 31 14:48:26.771016 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:26.770991    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod988269ba_5182_484e_ad71_c72e50c2fe3b.slice/crio-19d2c0a31a8524cb2c1820e3da25568d00f01a5f50b873f4a58a340cbb1e0a67 WatchSource:0}: Error finding container 19d2c0a31a8524cb2c1820e3da25568d00f01a5f50b873f4a58a340cbb1e0a67: Status 404 returned error can't find the container with id 19d2c0a31a8524cb2c1820e3da25568d00f01a5f50b873f4a58a340cbb1e0a67
Jul 31 14:48:26.778693 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.778662    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d947335a-f9d7-4fcd-b9e1-457477ab7f72-original-pull-secret\") pod \"global-pull-secret-syncer-nk8pw\" (UID: \"d947335a-f9d7-4fcd-b9e1-457477ab7f72\") " pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:26.778858 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:26.778840    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:26.778910 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:26.778904    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d947335a-f9d7-4fcd-b9e1-457477ab7f72-original-pull-secret podName:d947335a-f9d7-4fcd-b9e1-457477ab7f72 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:27.778886484 +0000 UTC m=+4.090444087 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d947335a-f9d7-4fcd-b9e1-457477ab7f72-original-pull-secret") pod "global-pull-secret-syncer-nk8pw" (UID: "d947335a-f9d7-4fcd-b9e1-457477ab7f72") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:26.793595 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:26.793439    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54adc061_d1ce_4ba7_83cf_3b2968693d52.slice/crio-976a3177efb4ddc6885f9f5da0a9b58396afd6546dd703937ceb0874debd8341 WatchSource:0}: Error finding container 976a3177efb4ddc6885f9f5da0a9b58396afd6546dd703937ceb0874debd8341: Status 404 returned error can't find the container with id 976a3177efb4ddc6885f9f5da0a9b58396afd6546dd703937ceb0874debd8341
Jul 31 14:48:26.794313 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:26.794288    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ce38fc_0695_4bfe_8292_3547c26c44d5.slice/crio-28f9fc994d0b2294a94bc917143c061a3de30b72ac04c92eacf1def9a5114b72 WatchSource:0}: Error finding container 28f9fc994d0b2294a94bc917143c061a3de30b72ac04c92eacf1def9a5114b72: Status 404 returned error can't find the container with id 28f9fc994d0b2294a94bc917143c061a3de30b72ac04c92eacf1def9a5114b72
Jul 31 14:48:26.816180 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:48:26.816154    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87bd827f8cab5b15b18522f993681f04.slice/crio-2251ea9ec0b4f4882c70f7914b1e6f96836d85073a58d95a224341eb6028341a WatchSource:0}: Error finding container 2251ea9ec0b4f4882c70f7914b1e6f96836d85073a58d95a224341eb6028341a: Status 404 returned error can't find the container with id 2251ea9ec0b4f4882c70f7914b1e6f96836d85073a58d95a224341eb6028341a
Jul 31 14:48:26.879843 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:26.879819    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97xf4\" (UniqueName: \"kubernetes.io/projected/567a8972-6aaf-499f-a8d8-a83da117da60-kube-api-access-97xf4\") pod \"network-check-target-6vfsr\" (UID: \"567a8972-6aaf-499f-a8d8-a83da117da60\") " pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:48:26.880013 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:26.879997    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:26.880056 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:26.880018    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:26.880056 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:26.880028    2559 projected.go:196] Error preparing data for projected volume kube-api-access-97xf4 for pod openshift-network-diagnostics/network-check-target-6vfsr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:26.880120 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:26.880086    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/567a8972-6aaf-499f-a8d8-a83da117da60-kube-api-access-97xf4 podName:567a8972-6aaf-499f-a8d8-a83da117da60 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:27.880069309 +0000 UTC m=+4.191626932 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-97xf4" (UniqueName: "kubernetes.io/projected/567a8972-6aaf-499f-a8d8-a83da117da60-kube-api-access-97xf4") pod "network-check-target-6vfsr" (UID: "567a8972-6aaf-499f-a8d8-a83da117da60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:27.209550 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:27.209442    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 14:43:26 +0000 UTC" deadline="2028-02-03 23:48:16.29057902 +0000 UTC"
Jul 31 14:48:27.209550 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:27.209484    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13256h59m49.081100057s"
Jul 31 14:48:27.227862 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:27.227814    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-219.ec2.internal" event={"ID":"13814aefbbec7fd254d31d3cf317ab12","Type":"ContainerStarted","Data":"ea4ad943fbd0eaa551ff8715aee770eb8bb3e5034bed7c864b910106d67b443a"}
Jul 31 14:48:27.230801 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:27.230703    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal" event={"ID":"87bd827f8cab5b15b18522f993681f04","Type":"ContainerStarted","Data":"2251ea9ec0b4f4882c70f7914b1e6f96836d85073a58d95a224341eb6028341a"}
Jul 31 14:48:27.232853 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:27.232812    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-64wqp" event={"ID":"88ce38fc-0695-4bfe-8292-3547c26c44d5","Type":"ContainerStarted","Data":"28f9fc994d0b2294a94bc917143c061a3de30b72ac04c92eacf1def9a5114b72"}
Jul 31 14:48:27.234672 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:27.234627    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xg6w7" event={"ID":"988269ba-5182-484e-ad71-c72e50c2fe3b","Type":"ContainerStarted","Data":"19d2c0a31a8524cb2c1820e3da25568d00f01a5f50b873f4a58a340cbb1e0a67"}
Jul 31 14:48:27.237165 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:27.237133    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd" event={"ID":"c55da51d-eba3-4993-a0bb-65f32e5d73cc","Type":"ContainerStarted","Data":"a48dacd1bda2284e4c0bd65cd22996f41a7ae6e49a2c9cc6728df467209184a6"}
Jul 31 14:48:27.239118 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:27.239013    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bwcjx" event={"ID":"16b5669e-0dde-49c0-854b-495854e99a82","Type":"ContainerStarted","Data":"4bec6b0cbd24ca34afd60d41f949af8c370f416117429aa361dde0322ed805c3"}
Jul 31 14:48:27.241668 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:27.241646    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xvpvq" event={"ID":"f6546deb-c60b-4bbb-9535-7dac6a479299","Type":"ContainerStarted","Data":"609e7259c447776087ba4ed17d508c1e98f7ca0137a31bf596d4bfc48d1c9f82"}
Jul 31 14:48:27.243274 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:27.243230    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mpkxl" event={"ID":"54adc061-d1ce-4ba7-83cf-3b2968693d52","Type":"ContainerStarted","Data":"976a3177efb4ddc6885f9f5da0a9b58396afd6546dd703937ceb0874debd8341"}
Jul 31 14:48:27.244819 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:27.244753    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jfrf7" event={"ID":"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24","Type":"ContainerStarted","Data":"c50c1b896513d703775de91e82d53d5c446d11bad736bb1d9c6b90375cbe6ad4"}
Jul 31 14:48:27.246647 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:27.246620    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tcn8w" event={"ID":"a734dd47-fffc-4dfd-8946-3c3466565b8c","Type":"ContainerStarted","Data":"d93aa454caa6088ab6d8c01bd9285f6d5b6cf73d0d90a23d56569c4bbb0b6b27"}
Jul 31 14:48:27.246792 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:27.246658    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-219.ec2.internal" podStartSLOduration=2.246639899 podStartE2EDuration="2.246639899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:27.245998023 +0000 UTC m=+3.557555651" watchObservedRunningTime="2026-07-31 14:48:27.246639899 +0000 UTC m=+3.558197528"
Jul 31 14:48:27.248693 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:27.248664    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57" event={"ID":"4c2a7673-4169-4edb-a234-70c7d12e0804","Type":"ContainerStarted","Data":"ebeecbdd65a9073ee88504396ce9003b744d8c014eb59a71cf3ded8c42b8e253"}
Jul 31 14:48:27.687886 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:27.687734    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355d9cbb-1867-49cc-8ec2-564b4aa6daea-metrics-certs\") pod \"network-metrics-daemon-b45r4\" (UID: \"355d9cbb-1867-49cc-8ec2-564b4aa6daea\") " pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:27.688076 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:27.687935    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:27.688076 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:27.688013    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/355d9cbb-1867-49cc-8ec2-564b4aa6daea-metrics-certs podName:355d9cbb-1867-49cc-8ec2-564b4aa6daea nodeName:}" failed. No retries permitted until 2026-07-31 14:48:29.687990673 +0000 UTC m=+5.999548280 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/355d9cbb-1867-49cc-8ec2-564b4aa6daea-metrics-certs") pod "network-metrics-daemon-b45r4" (UID: "355d9cbb-1867-49cc-8ec2-564b4aa6daea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:27.788971 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:27.788930    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d947335a-f9d7-4fcd-b9e1-457477ab7f72-original-pull-secret\") pod \"global-pull-secret-syncer-nk8pw\" (UID: \"d947335a-f9d7-4fcd-b9e1-457477ab7f72\") " pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:27.789131 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:27.789106    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:27.789203 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:27.789173    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d947335a-f9d7-4fcd-b9e1-457477ab7f72-original-pull-secret podName:d947335a-f9d7-4fcd-b9e1-457477ab7f72 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:29.789153533 +0000 UTC m=+6.100711150 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d947335a-f9d7-4fcd-b9e1-457477ab7f72-original-pull-secret") pod "global-pull-secret-syncer-nk8pw" (UID: "d947335a-f9d7-4fcd-b9e1-457477ab7f72") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:27.890781 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:27.890048    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97xf4\" (UniqueName: \"kubernetes.io/projected/567a8972-6aaf-499f-a8d8-a83da117da60-kube-api-access-97xf4\") pod \"network-check-target-6vfsr\" (UID: \"567a8972-6aaf-499f-a8d8-a83da117da60\") " pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:48:27.890781 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:27.890294    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:27.890781 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:27.890318    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:27.890781 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:27.890333    2559 projected.go:196] Error preparing data for projected volume kube-api-access-97xf4 for pod openshift-network-diagnostics/network-check-target-6vfsr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:27.890781 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:27.890398    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/567a8972-6aaf-499f-a8d8-a83da117da60-kube-api-access-97xf4 podName:567a8972-6aaf-499f-a8d8-a83da117da60 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:29.890377616 +0000 UTC m=+6.201935233 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-97xf4" (UniqueName: "kubernetes.io/projected/567a8972-6aaf-499f-a8d8-a83da117da60-kube-api-access-97xf4") pod "network-check-target-6vfsr" (UID: "567a8972-6aaf-499f-a8d8-a83da117da60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:28.209994 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:28.209899    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 14:43:26 +0000 UTC" deadline="2028-02-03 14:18:58.766674345 +0000 UTC"
Jul 31 14:48:28.209994 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:28.209962    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13247h30m30.556716773s"
Jul 31 14:48:28.216268 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:28.216243    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:48:28.216398 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:28.216374    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vfsr" podUID="567a8972-6aaf-499f-a8d8-a83da117da60"
Jul 31 14:48:28.216849 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:28.216829    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:28.216957 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:28.216936    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk8pw" podUID="d947335a-f9d7-4fcd-b9e1-457477ab7f72"
Jul 31 14:48:28.217061 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:28.217047    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:28.217168 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:28.217149    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b45r4" podUID="355d9cbb-1867-49cc-8ec2-564b4aa6daea"
Jul 31 14:48:29.708695 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:29.708651    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355d9cbb-1867-49cc-8ec2-564b4aa6daea-metrics-certs\") pod \"network-metrics-daemon-b45r4\" (UID: \"355d9cbb-1867-49cc-8ec2-564b4aa6daea\") " pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:29.709361 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:29.708863    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:29.709361 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:29.708932    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/355d9cbb-1867-49cc-8ec2-564b4aa6daea-metrics-certs podName:355d9cbb-1867-49cc-8ec2-564b4aa6daea nodeName:}" failed. No retries permitted until 2026-07-31 14:48:33.708911417 +0000 UTC m=+10.020469022 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/355d9cbb-1867-49cc-8ec2-564b4aa6daea-metrics-certs") pod "network-metrics-daemon-b45r4" (UID: "355d9cbb-1867-49cc-8ec2-564b4aa6daea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:29.809927 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:29.809255    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d947335a-f9d7-4fcd-b9e1-457477ab7f72-original-pull-secret\") pod \"global-pull-secret-syncer-nk8pw\" (UID: \"d947335a-f9d7-4fcd-b9e1-457477ab7f72\") " pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:29.809927 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:29.809466    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:29.809927 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:29.809536    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d947335a-f9d7-4fcd-b9e1-457477ab7f72-original-pull-secret podName:d947335a-f9d7-4fcd-b9e1-457477ab7f72 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:33.809515995 +0000 UTC m=+10.121073614 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d947335a-f9d7-4fcd-b9e1-457477ab7f72-original-pull-secret") pod "global-pull-secret-syncer-nk8pw" (UID: "d947335a-f9d7-4fcd-b9e1-457477ab7f72") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:29.910658 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:29.909982    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97xf4\" (UniqueName: \"kubernetes.io/projected/567a8972-6aaf-499f-a8d8-a83da117da60-kube-api-access-97xf4\") pod \"network-check-target-6vfsr\" (UID: \"567a8972-6aaf-499f-a8d8-a83da117da60\") " pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:48:29.910658 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:29.910189    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:29.910658 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:29.910211    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:29.910658 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:29.910225    2559 projected.go:196] Error preparing data for projected volume kube-api-access-97xf4 for pod openshift-network-diagnostics/network-check-target-6vfsr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:29.910658 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:29.910290    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/567a8972-6aaf-499f-a8d8-a83da117da60-kube-api-access-97xf4 podName:567a8972-6aaf-499f-a8d8-a83da117da60 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:33.9102698 +0000 UTC m=+10.221827421 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-97xf4" (UniqueName: "kubernetes.io/projected/567a8972-6aaf-499f-a8d8-a83da117da60-kube-api-access-97xf4") pod "network-check-target-6vfsr" (UID: "567a8972-6aaf-499f-a8d8-a83da117da60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:30.219002 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:30.218971    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:30.219177 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:30.219155    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b45r4" podUID="355d9cbb-1867-49cc-8ec2-564b4aa6daea"
Jul 31 14:48:30.219579 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:30.219556    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:48:30.219701 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:30.219686    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vfsr" podUID="567a8972-6aaf-499f-a8d8-a83da117da60"
Jul 31 14:48:30.219768 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:30.219736    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:30.219844 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:30.219825    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk8pw" podUID="d947335a-f9d7-4fcd-b9e1-457477ab7f72"
Jul 31 14:48:32.218426 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:32.217959    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:32.218426 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:32.218115    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b45r4" podUID="355d9cbb-1867-49cc-8ec2-564b4aa6daea"
Jul 31 14:48:32.218426 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:32.218204    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:48:32.218426 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:32.218302    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vfsr" podUID="567a8972-6aaf-499f-a8d8-a83da117da60"
Jul 31 14:48:32.218426 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:32.218350    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:32.218426 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:32.218396    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk8pw" podUID="d947335a-f9d7-4fcd-b9e1-457477ab7f72"
Jul 31 14:48:33.745024 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:33.744984    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355d9cbb-1867-49cc-8ec2-564b4aa6daea-metrics-certs\") pod \"network-metrics-daemon-b45r4\" (UID: \"355d9cbb-1867-49cc-8ec2-564b4aa6daea\") " pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:33.745503 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:33.745148    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:33.745503 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:33.745225    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/355d9cbb-1867-49cc-8ec2-564b4aa6daea-metrics-certs podName:355d9cbb-1867-49cc-8ec2-564b4aa6daea nodeName:}" failed. No retries permitted until 2026-07-31 14:48:41.745204778 +0000 UTC m=+18.056762387 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/355d9cbb-1867-49cc-8ec2-564b4aa6daea-metrics-certs") pod "network-metrics-daemon-b45r4" (UID: "355d9cbb-1867-49cc-8ec2-564b4aa6daea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:33.845841 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:33.845791    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d947335a-f9d7-4fcd-b9e1-457477ab7f72-original-pull-secret\") pod \"global-pull-secret-syncer-nk8pw\" (UID: \"d947335a-f9d7-4fcd-b9e1-457477ab7f72\") " pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:33.846028 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:33.845946    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:33.846028 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:33.846011    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d947335a-f9d7-4fcd-b9e1-457477ab7f72-original-pull-secret podName:d947335a-f9d7-4fcd-b9e1-457477ab7f72 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:41.845992852 +0000 UTC m=+18.157550456 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d947335a-f9d7-4fcd-b9e1-457477ab7f72-original-pull-secret") pod "global-pull-secret-syncer-nk8pw" (UID: "d947335a-f9d7-4fcd-b9e1-457477ab7f72") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:33.946846 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:33.946804    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97xf4\" (UniqueName: \"kubernetes.io/projected/567a8972-6aaf-499f-a8d8-a83da117da60-kube-api-access-97xf4\") pod \"network-check-target-6vfsr\" (UID: \"567a8972-6aaf-499f-a8d8-a83da117da60\") " pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:48:33.947003 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:33.946992    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:33.947057 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:33.947014    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:33.947057 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:33.947027    2559 projected.go:196] Error preparing data for projected volume kube-api-access-97xf4 for pod openshift-network-diagnostics/network-check-target-6vfsr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:33.947131 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:33.947083    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/567a8972-6aaf-499f-a8d8-a83da117da60-kube-api-access-97xf4 podName:567a8972-6aaf-499f-a8d8-a83da117da60 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:41.947070473 +0000 UTC m=+18.258628087 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-97xf4" (UniqueName: "kubernetes.io/projected/567a8972-6aaf-499f-a8d8-a83da117da60-kube-api-access-97xf4") pod "network-check-target-6vfsr" (UID: "567a8972-6aaf-499f-a8d8-a83da117da60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:34.218446 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:34.218369    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:34.218605 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:34.218503    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b45r4" podUID="355d9cbb-1867-49cc-8ec2-564b4aa6daea"
Jul 31 14:48:34.218605 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:34.218567    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:48:34.218715 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:34.218629    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vfsr" podUID="567a8972-6aaf-499f-a8d8-a83da117da60"
Jul 31 14:48:34.218715 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:34.218673    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:34.218843 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:34.218760    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk8pw" podUID="d947335a-f9d7-4fcd-b9e1-457477ab7f72"
Jul 31 14:48:36.217314 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:36.216451    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:48:36.217314 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:36.216581    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vfsr" podUID="567a8972-6aaf-499f-a8d8-a83da117da60"
Jul 31 14:48:36.217314 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:36.216988    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:36.217314 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:36.217084    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk8pw" podUID="d947335a-f9d7-4fcd-b9e1-457477ab7f72"
Jul 31 14:48:36.217314 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:36.217177    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:36.217314 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:36.217279    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b45r4" podUID="355d9cbb-1867-49cc-8ec2-564b4aa6daea"
Jul 31 14:48:38.219577 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:38.219549    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:48:38.220049 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:38.219658    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vfsr" podUID="567a8972-6aaf-499f-a8d8-a83da117da60"
Jul 31 14:48:38.220049 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:38.219715    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:38.220049 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:38.219809    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk8pw" podUID="d947335a-f9d7-4fcd-b9e1-457477ab7f72"
Jul 31 14:48:38.220049 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:38.219851    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:38.220049 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:38.219906    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b45r4" podUID="355d9cbb-1867-49cc-8ec2-564b4aa6daea"
Jul 31 14:48:40.216352 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:40.216323    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:40.216806 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:40.216326    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:48:40.216806 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:40.216456    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b45r4" podUID="355d9cbb-1867-49cc-8ec2-564b4aa6daea"
Jul 31 14:48:40.216806 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:40.216511    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:40.216806 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:40.216610    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vfsr" podUID="567a8972-6aaf-499f-a8d8-a83da117da60"
Jul 31 14:48:40.216806 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:40.216724    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk8pw" podUID="d947335a-f9d7-4fcd-b9e1-457477ab7f72"
Jul 31 14:48:41.804499 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:41.804464    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355d9cbb-1867-49cc-8ec2-564b4aa6daea-metrics-certs\") pod \"network-metrics-daemon-b45r4\" (UID: \"355d9cbb-1867-49cc-8ec2-564b4aa6daea\") " pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:41.805048 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:41.804601    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:41.805048 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:41.804660    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/355d9cbb-1867-49cc-8ec2-564b4aa6daea-metrics-certs podName:355d9cbb-1867-49cc-8ec2-564b4aa6daea nodeName:}" failed. No retries permitted until 2026-07-31 14:48:57.804642555 +0000 UTC m=+34.116200156 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/355d9cbb-1867-49cc-8ec2-564b4aa6daea-metrics-certs") pod "network-metrics-daemon-b45r4" (UID: "355d9cbb-1867-49cc-8ec2-564b4aa6daea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:41.905697 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:41.905664    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d947335a-f9d7-4fcd-b9e1-457477ab7f72-original-pull-secret\") pod \"global-pull-secret-syncer-nk8pw\" (UID: \"d947335a-f9d7-4fcd-b9e1-457477ab7f72\") " pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:41.905910 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:41.905843    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:41.905974 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:41.905928    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d947335a-f9d7-4fcd-b9e1-457477ab7f72-original-pull-secret podName:d947335a-f9d7-4fcd-b9e1-457477ab7f72 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:57.905907252 +0000 UTC m=+34.217465064 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d947335a-f9d7-4fcd-b9e1-457477ab7f72-original-pull-secret") pod "global-pull-secret-syncer-nk8pw" (UID: "d947335a-f9d7-4fcd-b9e1-457477ab7f72") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:42.006746 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:42.006692    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97xf4\" (UniqueName: \"kubernetes.io/projected/567a8972-6aaf-499f-a8d8-a83da117da60-kube-api-access-97xf4\") pod \"network-check-target-6vfsr\" (UID: \"567a8972-6aaf-499f-a8d8-a83da117da60\") " pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:48:42.006918 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:42.006851    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:42.006918 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:42.006870    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:42.006918 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:42.006883    2559 projected.go:196] Error preparing data for projected volume kube-api-access-97xf4 for pod openshift-network-diagnostics/network-check-target-6vfsr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:42.007049 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:42.006941    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/567a8972-6aaf-499f-a8d8-a83da117da60-kube-api-access-97xf4 podName:567a8972-6aaf-499f-a8d8-a83da117da60 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:58.006924955 +0000 UTC m=+34.318482576 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-97xf4" (UniqueName: "kubernetes.io/projected/567a8972-6aaf-499f-a8d8-a83da117da60-kube-api-access-97xf4") pod "network-check-target-6vfsr" (UID: "567a8972-6aaf-499f-a8d8-a83da117da60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:42.217230 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:42.216645    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:42.217230 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:42.216658    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:42.217230 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:42.216811    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b45r4" podUID="355d9cbb-1867-49cc-8ec2-564b4aa6daea"
Jul 31 14:48:42.217230 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:42.216658    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:48:42.217230 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:42.216871    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk8pw" podUID="d947335a-f9d7-4fcd-b9e1-457477ab7f72"
Jul 31 14:48:42.217230 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:42.216905    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vfsr" podUID="567a8972-6aaf-499f-a8d8-a83da117da60"
Jul 31 14:48:44.217194 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:44.217160    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:44.217661 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:44.217261    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk8pw" podUID="d947335a-f9d7-4fcd-b9e1-457477ab7f72"
Jul 31 14:48:44.217661 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:44.217343    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:48:44.217661 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:44.217433    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:44.217661 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:44.217463    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vfsr" podUID="567a8972-6aaf-499f-a8d8-a83da117da60"
Jul 31 14:48:44.217661 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:44.217501    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b45r4" podUID="355d9cbb-1867-49cc-8ec2-564b4aa6daea"
Jul 31 14:48:46.215957 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:46.215922    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:48:46.216594 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:46.216052    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vfsr" podUID="567a8972-6aaf-499f-a8d8-a83da117da60"
Jul 31 14:48:46.216594 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:46.216073    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:46.216594 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:46.216179    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b45r4" podUID="355d9cbb-1867-49cc-8ec2-564b4aa6daea"
Jul 31 14:48:46.216594 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:46.215924    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:46.216594 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:46.216262    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk8pw" podUID="d947335a-f9d7-4fcd-b9e1-457477ab7f72"
Jul 31 14:48:47.300478 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:47.300401    2559 generic.go:356] "Generic (PLEG): container finished" podID="87bd827f8cab5b15b18522f993681f04" containerID="018de2e4e0f5824d2976b1a9859a13a30f0f54734c097f4d417360acdfae94fb" exitCode=0
Jul 31 14:48:47.301347 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:47.300488    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal" event={"ID":"87bd827f8cab5b15b18522f993681f04","Type":"ContainerDied","Data":"018de2e4e0f5824d2976b1a9859a13a30f0f54734c097f4d417360acdfae94fb"}
Jul 31 14:48:47.302159 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:47.302139    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-64wqp" event={"ID":"88ce38fc-0695-4bfe-8292-3547c26c44d5","Type":"ContainerStarted","Data":"d2544408d18635ede32e2a4855f473649a5f81e711d603e18ff1f8209afa4874"}
Jul 31 14:48:47.303556 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:47.303531    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xg6w7" event={"ID":"988269ba-5182-484e-ad71-c72e50c2fe3b","Type":"ContainerStarted","Data":"6ff89e7b4a936530d5530039a32511f46990dfe53cd39835f3f8d2dfac1c44eb"}
Jul 31 14:48:47.306362 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:47.306342    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd" event={"ID":"c55da51d-eba3-4993-a0bb-65f32e5d73cc","Type":"ContainerStarted","Data":"eab139dbe34cd80d173b456fe4289c43a91c573cba4987fe9574699984a3cd31"}
Jul 31 14:48:47.306451 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:47.306399    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd" event={"ID":"c55da51d-eba3-4993-a0bb-65f32e5d73cc","Type":"ContainerStarted","Data":"f72b08454c497439628c62ec785bb243002cdbba7bc2b38686c0c7e8c76b11f7"}
Jul 31 14:48:47.306451 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:47.306438    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd" event={"ID":"c55da51d-eba3-4993-a0bb-65f32e5d73cc","Type":"ContainerStarted","Data":"9a2086152f472098b205b4101e8c206978ee94962652a0da658b47b30797b6a8"}
Jul 31 14:48:47.306522 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:47.306477    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd" event={"ID":"c55da51d-eba3-4993-a0bb-65f32e5d73cc","Type":"ContainerStarted","Data":"402fc6c7ed4d309eeb246825e55cd05ebac567a99ea518d3da3ada344a113e94"}
Jul 31 14:48:47.306522 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:47.306511    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd" event={"ID":"c55da51d-eba3-4993-a0bb-65f32e5d73cc","Type":"ContainerStarted","Data":"b12790ce6129c2bd92aaa8b69d1137131b15eff079529aa48e27bfc014d4f27a"}
Jul 31 14:48:47.306594 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:47.306540    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd" event={"ID":"c55da51d-eba3-4993-a0bb-65f32e5d73cc","Type":"ContainerStarted","Data":"cfa71c6ad57454ce0a810293ddcb0d4ef92666527054cce1be24082af3c26eba"}
Jul 31 14:48:47.307670 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:47.307647    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bwcjx" event={"ID":"16b5669e-0dde-49c0-854b-495854e99a82","Type":"ContainerStarted","Data":"cc7eff57bab7d975efeedeb7953407e96f6ad1d760b08baa37604fbd2f2ca44b"}
Jul 31 14:48:47.308977 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:47.308955    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xvpvq" event={"ID":"f6546deb-c60b-4bbb-9535-7dac6a479299","Type":"ContainerStarted","Data":"1daa759893a1e4c5153debe4740f6a0548be0ccc7f0df26fbc3dfd8dbe70bf1b"}
Jul 31 14:48:47.310284 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:47.310261    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mpkxl" event={"ID":"54adc061-d1ce-4ba7-83cf-3b2968693d52","Type":"ContainerStarted","Data":"ed89325700f1d4bf4a1573c1e872ae45cdbd2474be97ba54502c81ea3b076717"}
Jul 31 14:48:47.311614 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:47.311591    2559 generic.go:356] "Generic (PLEG): container finished" podID="38a6d9ec-ffc7-4d77-b8c2-819775c1cf24" containerID="4bc636bed17f6b35c5728fb541f76cd89ae962ecaa48e6baf3ab1f212ec883e5" exitCode=0
Jul 31 14:48:47.311698 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:47.311655    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jfrf7" event={"ID":"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24","Type":"ContainerDied","Data":"4bc636bed17f6b35c5728fb541f76cd89ae962ecaa48e6baf3ab1f212ec883e5"}
Jul 31 14:48:47.313113 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:47.313094    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57" event={"ID":"4c2a7673-4169-4edb-a234-70c7d12e0804","Type":"ContainerStarted","Data":"7473ab2f9ec4ee8195d8a1c73cd2d2693465ba33f3f0743ac7522c9dd2a90488"}
Jul 31 14:48:47.343230 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:47.343179    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xg6w7" podStartSLOduration=7.680115712 podStartE2EDuration="23.34316044s" totalImagesPullingTime="15.663044728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:47.343042667 +0000 UTC m=+23.654600289" watchObservedRunningTime="2026-07-31 14:48:47.34316044 +0000 UTC m=+23.654718063"
Jul 31 14:48:47.437821 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:47.437617    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xvpvq" podStartSLOduration=3.66370054 podStartE2EDuration="23.437601481s" totalImagesPullingTime="19.773900941s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:47.407940854 +0000 UTC m=+23.719498475" watchObservedRunningTime="2026-07-31 14:48:47.437601481 +0000 UTC m=+23.749159102"
Jul 31 14:48:47.489274 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:47.489223    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mpkxl" podStartSLOduration=3.777080203 podStartE2EDuration="23.489205861s" totalImagesPullingTime="19.712125658s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:47.437904875 +0000 UTC m=+23.749462487" watchObservedRunningTime="2026-07-31 14:48:47.489205861 +0000 UTC m=+23.800763482"
Jul 31 14:48:47.532610 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:47.532565    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bwcjx" podStartSLOduration=3.791091222 podStartE2EDuration="23.532547664s" totalImagesPullingTime="19.741456442s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:47.48935826 +0000 UTC m=+23.800915893" watchObservedRunningTime="2026-07-31 14:48:47.532547664 +0000 UTC m=+23.844105692"
Jul 31 14:48:48.216663 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:48.216627    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:48.216880 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:48.216805    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk8pw" podUID="d947335a-f9d7-4fcd-b9e1-457477ab7f72"
Jul 31 14:48:48.216880 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:48.216627    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:48:48.216997 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:48.216891    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vfsr" podUID="567a8972-6aaf-499f-a8d8-a83da117da60"
Jul 31 14:48:48.216997 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:48.216627    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:48.216997 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:48.216955    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b45r4" podUID="355d9cbb-1867-49cc-8ec2-564b4aa6daea"
Jul 31 14:48:48.320301 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:48.320260    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tcn8w" event={"ID":"a734dd47-fffc-4dfd-8946-3c3466565b8c","Type":"ContainerStarted","Data":"3a042f8612d93215e54e6eb528781634b85ca08f40388db1cb0d87e2b4448677"}
Jul 31 14:48:48.322567 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:48.322533    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal" event={"ID":"87bd827f8cab5b15b18522f993681f04","Type":"ContainerStarted","Data":"fcf6846136fae9d75a8fb490cab48b935ae0b083074e0418b413f5baa581a0ae"}
Jul 31 14:48:48.346040 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:48.345983    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-64wqp" podStartSLOduration=4.572789683 podStartE2EDuration="24.34594205s" totalImagesPullingTime="19.773152367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:47.539474116 +0000 UTC m=+23.851031732" watchObservedRunningTime="2026-07-31 14:48:48.34594205 +0000 UTC m=+24.657499673"
Jul 31 14:48:48.381915 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:48.381856    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tcn8w" podStartSLOduration=4.635144704 podStartE2EDuration="24.381838209s" totalImagesPullingTime="19.746693505s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:48.381177696 +0000 UTC m=+24.692735319" watchObservedRunningTime="2026-07-31 14:48:48.381838209 +0000 UTC m=+24.693395832"
Jul 31 14:48:48.382279 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:48.382252    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-219.ec2.internal" podStartSLOduration=23.382242918 podStartE2EDuration="23.382242918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:48.346705757 +0000 UTC m=+24.658263380" watchObservedRunningTime="2026-07-31 14:48:48.382242918 +0000 UTC m=+24.693800539"
Jul 31 14:48:48.441518 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:48.441491    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 31 14:48:49.135459 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:49.135365    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-31T14:48:48.441513577Z","UUID":"ea9ad788-b174-4a63-bade-a8d1c7a2de52","Handler":null,"Name":"","Endpoint":""}
Jul 31 14:48:49.136968 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:49.136934    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 31 14:48:49.137092 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:49.136996    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 31 14:48:49.327396 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:49.327360    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57" event={"ID":"4c2a7673-4169-4edb-a234-70c7d12e0804","Type":"ContainerStarted","Data":"8777fbf924a37bece534bf82365c605e76668792299f40138f1d1f10856f1771"}
Jul 31 14:48:50.216054 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:50.216012    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:48:50.216300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:50.216070    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:50.216300 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:50.216145    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vfsr" podUID="567a8972-6aaf-499f-a8d8-a83da117da60"
Jul 31 14:48:50.216300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:50.216012    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:50.216300 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:50.216254    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b45r4" podUID="355d9cbb-1867-49cc-8ec2-564b4aa6daea"
Jul 31 14:48:50.216442 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:50.216307    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk8pw" podUID="d947335a-f9d7-4fcd-b9e1-457477ab7f72"
Jul 31 14:48:50.221460 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:50.221432    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xg6w7"
Jul 31 14:48:50.222141 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:50.222123    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xg6w7"
Jul 31 14:48:50.334145 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:50.334104    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd" event={"ID":"c55da51d-eba3-4993-a0bb-65f32e5d73cc","Type":"ContainerStarted","Data":"e014104ad4d57635ab9318928c392497a296611772ba7caa5f6e742fc31d5ee5"}
Jul 31 14:48:50.336462 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:50.336432    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57" event={"ID":"4c2a7673-4169-4edb-a234-70c7d12e0804","Type":"ContainerStarted","Data":"8c9e81df1b95789422d5a38c0b86fcc9f551b11dde2f33f292d4cf9eef589d39"}
Jul 31 14:48:50.336655 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:50.336636    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xg6w7"
Jul 31 14:48:50.337163 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:50.337146    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xg6w7"
Jul 31 14:48:50.363362 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:50.363312    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmw57" podStartSLOduration=4.169722135 podStartE2EDuration="26.363291075s" totalImagesPullingTime="22.19356894s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:50.36329255 +0000 UTC m=+26.674850193" watchObservedRunningTime="2026-07-31 14:48:50.363291075 +0000 UTC m=+26.674848699"
Jul 31 14:48:52.216407 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:52.216162    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:52.217184 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:52.216163    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:52.217184 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:52.216453    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk8pw" podUID="d947335a-f9d7-4fcd-b9e1-457477ab7f72"
Jul 31 14:48:52.217184 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:52.216547    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b45r4" podUID="355d9cbb-1867-49cc-8ec2-564b4aa6daea"
Jul 31 14:48:52.217184 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:52.216165    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:48:52.217184 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:52.216642    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vfsr" podUID="567a8972-6aaf-499f-a8d8-a83da117da60"
Jul 31 14:48:52.343666 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:52.343630    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd" event={"ID":"c55da51d-eba3-4993-a0bb-65f32e5d73cc","Type":"ContainerStarted","Data":"c6a5ee0e8455352db7ce7addb627c70853b69f41004d2edfde6dc27f85a0cc14"}
Jul 31 14:48:52.343870 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:52.343842    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:52.345442 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:52.345415    2559 generic.go:356] "Generic (PLEG): container finished" podID="38a6d9ec-ffc7-4d77-b8c2-819775c1cf24" containerID="b3a724eee3d48f5be940ca5968156d2b2063347674bacfa928bb2eea6c172788" exitCode=0
Jul 31 14:48:52.345569 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:52.345468    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jfrf7" event={"ID":"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24","Type":"ContainerDied","Data":"b3a724eee3d48f5be940ca5968156d2b2063347674bacfa928bb2eea6c172788"}
Jul 31 14:48:52.360401 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:52.360375    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:52.375625 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:52.375580    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd" podStartSLOduration=8.588224708 podStartE2EDuration="28.375566317s" totalImagesPullingTime="19.787341609s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:52.375041966 +0000 UTC m=+28.686599618" watchObservedRunningTime="2026-07-31 14:48:52.375566317 +0000 UTC m=+28.687123938"
Jul 31 14:48:53.348064 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:53.348035    2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 31 14:48:53.348805 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:53.348557    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:53.364744 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:53.364700    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:53.856072 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:53.855873    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nk8pw"]
Jul 31 14:48:53.856264 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:53.856166    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:53.856309 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:53.856282    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk8pw" podUID="d947335a-f9d7-4fcd-b9e1-457477ab7f72"
Jul 31 14:48:53.859223 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:53.859186    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6vfsr"]
Jul 31 14:48:53.859348 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:53.859334    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:48:53.859482 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:53.859457    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vfsr" podUID="567a8972-6aaf-499f-a8d8-a83da117da60"
Jul 31 14:48:53.859747 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:53.859705    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b45r4"]
Jul 31 14:48:53.859852 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:53.859837    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:53.859954 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:53.859933    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b45r4" podUID="355d9cbb-1867-49cc-8ec2-564b4aa6daea"
Jul 31 14:48:54.351964 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:54.351931    2559 generic.go:356] "Generic (PLEG): container finished" podID="38a6d9ec-ffc7-4d77-b8c2-819775c1cf24" containerID="79e683e6464335c6a8a50cbeea12e73e90314cb771f4ddc1ba59d34d3c3cb501" exitCode=0
Jul 31 14:48:54.352333 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:54.352017    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jfrf7" event={"ID":"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24","Type":"ContainerDied","Data":"79e683e6464335c6a8a50cbeea12e73e90314cb771f4ddc1ba59d34d3c3cb501"}
Jul 31 14:48:54.352333 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:54.352069    2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 31 14:48:55.215715 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:55.215682    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:55.215945 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:55.215744    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:48:55.215945 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:55.215832    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b45r4" podUID="355d9cbb-1867-49cc-8ec2-564b4aa6daea"
Jul 31 14:48:55.215945 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:55.215874    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vfsr" podUID="567a8972-6aaf-499f-a8d8-a83da117da60"
Jul 31 14:48:55.354404 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:55.354382    2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 31 14:48:56.071648 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:56.071606    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:48:56.087486 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:56.087441    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd" podUID="c55da51d-eba3-4993-a0bb-65f32e5d73cc" containerName="ovnkube-controller" probeResult="failure" output=""
Jul 31 14:48:56.096563 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:56.096525    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd" podUID="c55da51d-eba3-4993-a0bb-65f32e5d73cc" containerName="ovnkube-controller" probeResult="failure" output=""
Jul 31 14:48:56.219265 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:56.219200    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:56.219391 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:56.219298    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk8pw" podUID="d947335a-f9d7-4fcd-b9e1-457477ab7f72"
Jul 31 14:48:56.358846 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:56.358812    2559 generic.go:356] "Generic (PLEG): container finished" podID="38a6d9ec-ffc7-4d77-b8c2-819775c1cf24" containerID="896e0629e685de6129d35e50889e38e8ffe3cd89955ae5fa265c6ca71f896677" exitCode=0
Jul 31 14:48:56.359251 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:56.358886    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jfrf7" event={"ID":"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24","Type":"ContainerDied","Data":"896e0629e685de6129d35e50889e38e8ffe3cd89955ae5fa265c6ca71f896677"}
Jul 31 14:48:56.369508 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:56.369479    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd" podUID="c55da51d-eba3-4993-a0bb-65f32e5d73cc" containerName="ovnkube-controller" probeResult="failure" output=""
Jul 31 14:48:57.216223 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:57.216188    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:48:57.216412 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:57.216193    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:57.216412 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:57.216312    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vfsr" podUID="567a8972-6aaf-499f-a8d8-a83da117da60"
Jul 31 14:48:57.216412 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:57.216404    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b45r4" podUID="355d9cbb-1867-49cc-8ec2-564b4aa6daea"
Jul 31 14:48:57.825847 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:57.825761    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355d9cbb-1867-49cc-8ec2-564b4aa6daea-metrics-certs\") pod \"network-metrics-daemon-b45r4\" (UID: \"355d9cbb-1867-49cc-8ec2-564b4aa6daea\") " pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:57.826275 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:57.825909    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:57.826275 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:57.825972    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/355d9cbb-1867-49cc-8ec2-564b4aa6daea-metrics-certs podName:355d9cbb-1867-49cc-8ec2-564b4aa6daea nodeName:}" failed. No retries permitted until 2026-07-31 14:49:29.825952854 +0000 UTC m=+66.137510470 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/355d9cbb-1867-49cc-8ec2-564b4aa6daea-metrics-certs") pod "network-metrics-daemon-b45r4" (UID: "355d9cbb-1867-49cc-8ec2-564b4aa6daea") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:57.926550 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:57.926333    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d947335a-f9d7-4fcd-b9e1-457477ab7f72-original-pull-secret\") pod \"global-pull-secret-syncer-nk8pw\" (UID: \"d947335a-f9d7-4fcd-b9e1-457477ab7f72\") " pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:57.926705 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:57.926481    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:57.926705 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:57.926678    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d947335a-f9d7-4fcd-b9e1-457477ab7f72-original-pull-secret podName:d947335a-f9d7-4fcd-b9e1-457477ab7f72 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:29.926658362 +0000 UTC m=+66.238215979 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d947335a-f9d7-4fcd-b9e1-457477ab7f72-original-pull-secret") pod "global-pull-secret-syncer-nk8pw" (UID: "d947335a-f9d7-4fcd-b9e1-457477ab7f72") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:58.027515 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:58.027478    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97xf4\" (UniqueName: \"kubernetes.io/projected/567a8972-6aaf-499f-a8d8-a83da117da60-kube-api-access-97xf4\") pod \"network-check-target-6vfsr\" (UID: \"567a8972-6aaf-499f-a8d8-a83da117da60\") " pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:48:58.027695 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:58.027661    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:58.027695 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:58.027686    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:58.027695 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:58.027698    2559 projected.go:196] Error preparing data for projected volume kube-api-access-97xf4 for pod openshift-network-diagnostics/network-check-target-6vfsr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:58.027834 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:58.027767    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/567a8972-6aaf-499f-a8d8-a83da117da60-kube-api-access-97xf4 podName:567a8972-6aaf-499f-a8d8-a83da117da60 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:30.027750579 +0000 UTC m=+66.339308181 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-97xf4" (UniqueName: "kubernetes.io/projected/567a8972-6aaf-499f-a8d8-a83da117da60-kube-api-access-97xf4") pod "network-check-target-6vfsr" (UID: "567a8972-6aaf-499f-a8d8-a83da117da60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:58.219915 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:58.219844    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:48:58.220061 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:58.219966    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nk8pw" podUID="d947335a-f9d7-4fcd-b9e1-457477ab7f72"
Jul 31 14:48:59.215998 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.215959    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:48:59.216498 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.215964    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:48:59.216498 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:59.216096    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vfsr" podUID="567a8972-6aaf-499f-a8d8-a83da117da60"
Jul 31 14:48:59.216498 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:59.216170    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b45r4" podUID="355d9cbb-1867-49cc-8ec2-564b4aa6daea"
Jul 31 14:48:59.488402 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.488373    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-219.ec2.internal" event="NodeReady"
Jul 31 14:48:59.488571 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.488521    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 31 14:48:59.540698 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.540664    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2mr24"]
Jul 31 14:48:59.543554 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.543532    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2mr24"
Jul 31 14:48:59.545993 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.545957    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 31 14:48:59.546119 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.546100    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 31 14:48:59.546211 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.546187    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-plcv4\""
Jul 31 14:48:59.550295 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.550274    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fz26d"]
Jul 31 14:48:59.553053 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.552622    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fz26d"
Jul 31 14:48:59.555848 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.555586    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 31 14:48:59.555848 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.555657    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 31 14:48:59.556084 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.556069    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 31 14:48:59.556213 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.556196    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-p85t9\""
Jul 31 14:48:59.556341 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.556324    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2mr24"]
Jul 31 14:48:59.562199 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.562172    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fz26d"]
Jul 31 14:48:59.639671 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.639637    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee553c57-d835-4598-a0ee-b6fe7d685490-metrics-tls\") pod \"dns-default-2mr24\" (UID: \"ee553c57-d835-4598-a0ee-b6fe7d685490\") " pod="openshift-dns/dns-default-2mr24"
Jul 31 14:48:59.639855 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.639706    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kclj\" (UniqueName: \"kubernetes.io/projected/ba20a352-de91-49e8-a2ac-b5e1b9c37ae4-kube-api-access-7kclj\") pod \"ingress-canary-fz26d\" (UID: \"ba20a352-de91-49e8-a2ac-b5e1b9c37ae4\") " pod="openshift-ingress-canary/ingress-canary-fz26d"
Jul 31 14:48:59.639855 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.639744    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ee553c57-d835-4598-a0ee-b6fe7d685490-config-volume\") pod \"dns-default-2mr24\" (UID: \"ee553c57-d835-4598-a0ee-b6fe7d685490\") " pod="openshift-dns/dns-default-2mr24"
Jul 31 14:48:59.639855 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.639781    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdw74\" (UniqueName: \"kubernetes.io/projected/ee553c57-d835-4598-a0ee-b6fe7d685490-kube-api-access-pdw74\") pod \"dns-default-2mr24\" (UID: \"ee553c57-d835-4598-a0ee-b6fe7d685490\") " pod="openshift-dns/dns-default-2mr24"
Jul 31 14:48:59.639855 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.639799    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee553c57-d835-4598-a0ee-b6fe7d685490-tmp-dir\") pod \"dns-default-2mr24\" (UID: \"ee553c57-d835-4598-a0ee-b6fe7d685490\") " pod="openshift-dns/dns-default-2mr24"
Jul 31 14:48:59.639855 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.639821    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba20a352-de91-49e8-a2ac-b5e1b9c37ae4-cert\") pod \"ingress-canary-fz26d\" (UID: \"ba20a352-de91-49e8-a2ac-b5e1b9c37ae4\") " pod="openshift-ingress-canary/ingress-canary-fz26d"
Jul 31 14:48:59.741079 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.740987    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee553c57-d835-4598-a0ee-b6fe7d685490-metrics-tls\") pod \"dns-default-2mr24\" (UID: \"ee553c57-d835-4598-a0ee-b6fe7d685490\") " pod="openshift-dns/dns-default-2mr24"
Jul 31 14:48:59.741262 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.741080    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kclj\" (UniqueName: \"kubernetes.io/projected/ba20a352-de91-49e8-a2ac-b5e1b9c37ae4-kube-api-access-7kclj\") pod \"ingress-canary-fz26d\" (UID: \"ba20a352-de91-49e8-a2ac-b5e1b9c37ae4\") " pod="openshift-ingress-canary/ingress-canary-fz26d"
Jul 31 14:48:59.741262 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.741104    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ee553c57-d835-4598-a0ee-b6fe7d685490-config-volume\") pod \"dns-default-2mr24\" (UID: \"ee553c57-d835-4598-a0ee-b6fe7d685490\") " pod="openshift-dns/dns-default-2mr24"
Jul 31 14:48:59.741262 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.741129    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdw74\" (UniqueName: \"kubernetes.io/projected/ee553c57-d835-4598-a0ee-b6fe7d685490-kube-api-access-pdw74\") pod \"dns-default-2mr24\" (UID: \"ee553c57-d835-4598-a0ee-b6fe7d685490\") " pod="openshift-dns/dns-default-2mr24"
Jul 31 14:48:59.741262 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.741154    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee553c57-d835-4598-a0ee-b6fe7d685490-tmp-dir\") pod \"dns-default-2mr24\" (UID: \"ee553c57-d835-4598-a0ee-b6fe7d685490\") " pod="openshift-dns/dns-default-2mr24"
Jul 31 14:48:59.741262 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:59.741129    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:48:59.741262 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.741183    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba20a352-de91-49e8-a2ac-b5e1b9c37ae4-cert\") pod \"ingress-canary-fz26d\" (UID: \"ba20a352-de91-49e8-a2ac-b5e1b9c37ae4\") " pod="openshift-ingress-canary/ingress-canary-fz26d"
Jul 31 14:48:59.741538 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:48:59.741267    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee553c57-d835-4598-a0ee-b6fe7d685490-metrics-tls podName:ee553c57-d835-4598-a0ee-b6fe7d685490 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:00.241241162 +0000 UTC m=+36.552798766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ee553c57-d835-4598-a0ee-b6fe7d685490-metrics-tls") pod "dns-default-2mr24" (UID: "ee553c57-d835-4598-a0ee-b6fe7d685490") : secret "dns-default-metrics-tls" not found
Jul 31 14:48:59.741638 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.741606    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee553c57-d835-4598-a0ee-b6fe7d685490-tmp-dir\") pod \"dns-default-2mr24\" (UID: \"ee553c57-d835-4598-a0ee-b6fe7d685490\") " pod="openshift-dns/dns-default-2mr24"
Jul 31 14:48:59.741820 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.741795    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ee553c57-d835-4598-a0ee-b6fe7d685490-config-volume\") pod \"dns-default-2mr24\" (UID: \"ee553c57-d835-4598-a0ee-b6fe7d685490\") " pod="openshift-dns/dns-default-2mr24"
Jul 31 14:48:59.746447 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.746421    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba20a352-de91-49e8-a2ac-b5e1b9c37ae4-cert\") pod \"ingress-canary-fz26d\" (UID: \"ba20a352-de91-49e8-a2ac-b5e1b9c37ae4\") " pod="openshift-ingress-canary/ingress-canary-fz26d"
Jul 31 14:48:59.763520 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.763454    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdw74\" (UniqueName: \"kubernetes.io/projected/ee553c57-d835-4598-a0ee-b6fe7d685490-kube-api-access-pdw74\") pod \"dns-default-2mr24\" (UID: \"ee553c57-d835-4598-a0ee-b6fe7d685490\") " pod="openshift-dns/dns-default-2mr24"
Jul 31 14:48:59.765229 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.765198    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kclj\" (UniqueName: \"kubernetes.io/projected/ba20a352-de91-49e8-a2ac-b5e1b9c37ae4-kube-api-access-7kclj\") pod \"ingress-canary-fz26d\" (UID: \"ba20a352-de91-49e8-a2ac-b5e1b9c37ae4\") " pod="openshift-ingress-canary/ingress-canary-fz26d"
Jul 31 14:48:59.863856 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:48:59.863823    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fz26d"
Jul 31 14:49:00.216300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:00.216268    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:49:00.220942 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:00.220913    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 14:49:00.245023 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:00.244973    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee553c57-d835-4598-a0ee-b6fe7d685490-metrics-tls\") pod \"dns-default-2mr24\" (UID: \"ee553c57-d835-4598-a0ee-b6fe7d685490\") " pod="openshift-dns/dns-default-2mr24"
Jul 31 14:49:00.245170 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:00.245129    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:49:00.245231 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:00.245186    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee553c57-d835-4598-a0ee-b6fe7d685490-metrics-tls podName:ee553c57-d835-4598-a0ee-b6fe7d685490 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:01.245170159 +0000 UTC m=+37.556727776 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ee553c57-d835-4598-a0ee-b6fe7d685490-metrics-tls") pod "dns-default-2mr24" (UID: "ee553c57-d835-4598-a0ee-b6fe7d685490") : secret "dns-default-metrics-tls" not found
Jul 31 14:49:00.453871 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:00.453607    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fz26d"]
Jul 31 14:49:00.455127 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:49:00.455102    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba20a352_de91_49e8_a2ac_b5e1b9c37ae4.slice/crio-9a2f061810ece780d5d8de5864b6688e59ef511b9d30f7e8d291c52d859adc27 WatchSource:0}: Error finding container 9a2f061810ece780d5d8de5864b6688e59ef511b9d30f7e8d291c52d859adc27: Status 404 returned error can't find the container with id 9a2f061810ece780d5d8de5864b6688e59ef511b9d30f7e8d291c52d859adc27
Jul 31 14:49:01.216571 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:01.216532    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:49:01.217371 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:01.216545    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:49:01.220245 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:01.220220    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wtnft\""
Jul 31 14:49:01.220391 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:01.220254    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 14:49:01.220391 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:01.220308    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 14:49:01.220391 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:01.220324    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jrzg7\""
Jul 31 14:49:01.220523 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:01.220481    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 14:49:01.252806 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:01.252775    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee553c57-d835-4598-a0ee-b6fe7d685490-metrics-tls\") pod \"dns-default-2mr24\" (UID: \"ee553c57-d835-4598-a0ee-b6fe7d685490\") " pod="openshift-dns/dns-default-2mr24"
Jul 31 14:49:01.252967 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:01.252906    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:49:01.252967 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:01.252965    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee553c57-d835-4598-a0ee-b6fe7d685490-metrics-tls podName:ee553c57-d835-4598-a0ee-b6fe7d685490 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:03.252947729 +0000 UTC m=+39.564505352 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ee553c57-d835-4598-a0ee-b6fe7d685490-metrics-tls") pod "dns-default-2mr24" (UID: "ee553c57-d835-4598-a0ee-b6fe7d685490") : secret "dns-default-metrics-tls" not found
Jul 31 14:49:01.374325 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:01.374287    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fz26d" event={"ID":"ba20a352-de91-49e8-a2ac-b5e1b9c37ae4","Type":"ContainerStarted","Data":"9a2f061810ece780d5d8de5864b6688e59ef511b9d30f7e8d291c52d859adc27"}
Jul 31 14:49:01.377599 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:01.377567    2559 generic.go:356] "Generic (PLEG): container finished" podID="38a6d9ec-ffc7-4d77-b8c2-819775c1cf24" containerID="b043dca6be90b5c8e5fcac607110f240a1d204ed6fd8b7fdc06c4ce613386db0" exitCode=0
Jul 31 14:49:01.377765 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:01.377617    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jfrf7" event={"ID":"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24","Type":"ContainerDied","Data":"b043dca6be90b5c8e5fcac607110f240a1d204ed6fd8b7fdc06c4ce613386db0"}
Jul 31 14:49:02.384108 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:02.384073    2559 generic.go:356] "Generic (PLEG): container finished" podID="38a6d9ec-ffc7-4d77-b8c2-819775c1cf24" containerID="c08cf15470c08d0ea6aeb53855b11bcaa304c435f22fef374a0efbab0980df46" exitCode=0
Jul 31 14:49:02.384498 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:02.384117    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jfrf7" event={"ID":"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24","Type":"ContainerDied","Data":"c08cf15470c08d0ea6aeb53855b11bcaa304c435f22fef374a0efbab0980df46"}
Jul 31 14:49:03.268477 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:03.268439    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee553c57-d835-4598-a0ee-b6fe7d685490-metrics-tls\") pod \"dns-default-2mr24\" (UID: \"ee553c57-d835-4598-a0ee-b6fe7d685490\") " pod="openshift-dns/dns-default-2mr24"
Jul 31 14:49:03.268648 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:03.268546    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:49:03.268648 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:03.268598    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee553c57-d835-4598-a0ee-b6fe7d685490-metrics-tls podName:ee553c57-d835-4598-a0ee-b6fe7d685490 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:07.268585352 +0000 UTC m=+43.580142953 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ee553c57-d835-4598-a0ee-b6fe7d685490-metrics-tls") pod "dns-default-2mr24" (UID: "ee553c57-d835-4598-a0ee-b6fe7d685490") : secret "dns-default-metrics-tls" not found
Jul 31 14:49:03.388082 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:03.387855    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fz26d" event={"ID":"ba20a352-de91-49e8-a2ac-b5e1b9c37ae4","Type":"ContainerStarted","Data":"6643a41ee3406f32c9bbe8f9d3985ac69f5d4e86df57ea36d40951f62fb34ff6"}
Jul 31 14:49:03.391084 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:03.391059    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jfrf7" event={"ID":"38a6d9ec-ffc7-4d77-b8c2-819775c1cf24","Type":"ContainerStarted","Data":"7186b1cb45d98c760b62f62d4bf115e9413957abf8cc3d19a86e0adce7b3fe2d"}
Jul 31 14:49:03.414640 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:03.414596    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fz26d" podStartSLOduration=2.147942468 podStartE2EDuration="4.41457974s" totalImagesPullingTime="2.266637272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:03.414409772 +0000 UTC m=+39.725967394" watchObservedRunningTime="2026-07-31 14:49:03.41457974 +0000 UTC m=+39.726137379"
Jul 31 14:49:03.442058 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:03.441935    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jfrf7" podStartSLOduration=9.452092132 podStartE2EDuration="39.441914089s" totalImagesPullingTime="29.989821957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:03.441230748 +0000 UTC m=+39.752788370" watchObservedRunningTime="2026-07-31 14:49:03.441914089 +0000 UTC m=+39.753471713"
Jul 31 14:49:07.298828 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:07.298787    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee553c57-d835-4598-a0ee-b6fe7d685490-metrics-tls\") pod \"dns-default-2mr24\" (UID: \"ee553c57-d835-4598-a0ee-b6fe7d685490\") " pod="openshift-dns/dns-default-2mr24"
Jul 31 14:49:07.299209 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:07.298960    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:49:07.299209 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:07.299026    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee553c57-d835-4598-a0ee-b6fe7d685490-metrics-tls podName:ee553c57-d835-4598-a0ee-b6fe7d685490 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:15.299010073 +0000 UTC m=+51.610567674 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ee553c57-d835-4598-a0ee-b6fe7d685490-metrics-tls") pod "dns-default-2mr24" (UID: "ee553c57-d835-4598-a0ee-b6fe7d685490") : secret "dns-default-metrics-tls" not found
Jul 31 14:49:13.777158 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.777118    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m4hkx"]
Jul 31 14:49:13.785962 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.785933    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m4hkx"
Jul 31 14:49:13.788540 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.788516    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 31 14:49:13.789130 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.789106    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 31 14:49:13.789130 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.789125    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 31 14:49:13.791231 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.791210    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gc59x\""
Jul 31 14:49:13.791335 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.791210    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 31 14:49:13.794405 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.794384    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m4hkx"]
Jul 31 14:49:13.905068 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.905035    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-c6dsw"]
Jul 31 14:49:13.922996 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.922967    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-rgrwh"]
Jul 31 14:49:13.923142 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.923090    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-c6dsw"
Jul 31 14:49:13.925707 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.925682    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 31 14:49:13.926061 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.926036    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:49:13.926374 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.926358    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 31 14:49:13.926771 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.926752    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-n58dt\""
Jul 31 14:49:13.926901 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.926881    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 31 14:49:13.944559 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.944537    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-mngjp"]
Jul 31 14:49:13.944706 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.944690    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-rgrwh"
Jul 31 14:49:13.944772 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.944746    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e1cfecbe-4155-4b24-aad5-2277fd0c350e-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-m4hkx\" (UID: \"e1cfecbe-4155-4b24-aad5-2277fd0c350e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m4hkx"
Jul 31 14:49:13.944870 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.944839    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e1cfecbe-4155-4b24-aad5-2277fd0c350e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-m4hkx\" (UID: \"e1cfecbe-4155-4b24-aad5-2277fd0c350e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m4hkx"
Jul 31 14:49:13.944924 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.944905    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5knbp\" (UniqueName: \"kubernetes.io/projected/e1cfecbe-4155-4b24-aad5-2277fd0c350e-kube-api-access-5knbp\") pod \"cluster-monitoring-operator-6f55c4f58d-m4hkx\" (UID: \"e1cfecbe-4155-4b24-aad5-2277fd0c350e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m4hkx"
Jul 31 14:49:13.951489 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.951469    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 31 14:49:13.953883 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.953859    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-jv8hp\""
Jul 31 14:49:13.956837 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.956815    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s787l"]
Jul 31 14:49:13.956953 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.956938    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mngjp"
Jul 31 14:49:13.977593 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.977560    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:49:13.978491 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.978469    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 31 14:49:13.978491 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.978482    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 31 14:49:13.978660 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.978511    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 31 14:49:13.978660 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.978482    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:49:13.978660 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.978571    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-n7bmh\""
Jul 31 14:49:13.982616 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.982599    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-rgrwh"]
Jul 31 14:49:13.982697 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.982626    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s787l"]
Jul 31 14:49:13.982697 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.982643    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-c6dsw"]
Jul 31 14:49:13.982697 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.982657    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-mngjp"]
Jul 31 14:49:13.982807 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.982713    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s787l"
Jul 31 14:49:13.991035 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.991009    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-w52wl\""
Jul 31 14:49:13.991143 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.991106    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 31 14:49:13.991685 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.991670    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:49:13.993219 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:13.993206    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 31 14:49:14.045591 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.045508    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcdlm\" (UniqueName: \"kubernetes.io/projected/689c6bab-fc78-4cc4-81be-50463036b079-kube-api-access-fcdlm\") pod \"volume-data-source-validator-5b8776669-rgrwh\" (UID: \"689c6bab-fc78-4cc4-81be-50463036b079\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-rgrwh"
Jul 31 14:49:14.045591 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.045553    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/473e45a2-a84a-4a9f-8950-959e755baefa-serving-cert\") pod \"service-ca-operator-784f97bcf6-mngjp\" (UID: \"473e45a2-a84a-4a9f-8950-959e755baefa\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mngjp"
Jul 31 14:49:14.045591 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.045570    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1380fa66-45b6-48b5-aee6-18f70ca49ebf-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-c6dsw\" (UID: \"1380fa66-45b6-48b5-aee6-18f70ca49ebf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-c6dsw"
Jul 31 14:49:14.045811 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.045645    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e1cfecbe-4155-4b24-aad5-2277fd0c350e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-m4hkx\" (UID: \"e1cfecbe-4155-4b24-aad5-2277fd0c350e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m4hkx"
Jul 31 14:49:14.045811 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.045678    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5knbp\" (UniqueName: \"kubernetes.io/projected/e1cfecbe-4155-4b24-aad5-2277fd0c350e-kube-api-access-5knbp\") pod \"cluster-monitoring-operator-6f55c4f58d-m4hkx\" (UID: \"e1cfecbe-4155-4b24-aad5-2277fd0c350e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m4hkx"
Jul 31 14:49:14.045811 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.045713    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzlfm\" (UniqueName: \"kubernetes.io/projected/473e45a2-a84a-4a9f-8950-959e755baefa-kube-api-access-jzlfm\") pod \"service-ca-operator-784f97bcf6-mngjp\" (UID: \"473e45a2-a84a-4a9f-8950-959e755baefa\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mngjp"
Jul 31 14:49:14.045811 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.045754    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1380fa66-45b6-48b5-aee6-18f70ca49ebf-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-c6dsw\" (UID: \"1380fa66-45b6-48b5-aee6-18f70ca49ebf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-c6dsw"
Jul 31 14:49:14.045811 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.045796    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/473e45a2-a84a-4a9f-8950-959e755baefa-config\") pod \"service-ca-operator-784f97bcf6-mngjp\" (UID: \"473e45a2-a84a-4a9f-8950-959e755baefa\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mngjp"
Jul 31 14:49:14.045982 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:14.045814    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 14:49:14.045982 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.045824    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wsn7\" (UniqueName: \"kubernetes.io/projected/1380fa66-45b6-48b5-aee6-18f70ca49ebf-kube-api-access-9wsn7\") pod \"kube-storage-version-migrator-operator-7878f4b88c-c6dsw\" (UID: \"1380fa66-45b6-48b5-aee6-18f70ca49ebf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-c6dsw"
Jul 31 14:49:14.045982 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:14.045877    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1cfecbe-4155-4b24-aad5-2277fd0c350e-cluster-monitoring-operator-tls podName:e1cfecbe-4155-4b24-aad5-2277fd0c350e nodeName:}" failed. No retries permitted until 2026-07-31 14:49:14.545862096 +0000 UTC m=+50.857419695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e1cfecbe-4155-4b24-aad5-2277fd0c350e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-m4hkx" (UID: "e1cfecbe-4155-4b24-aad5-2277fd0c350e") : secret "cluster-monitoring-operator-tls" not found
Jul 31 14:49:14.045982 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.045934    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e1cfecbe-4155-4b24-aad5-2277fd0c350e-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-m4hkx\" (UID: \"e1cfecbe-4155-4b24-aad5-2277fd0c350e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m4hkx"
Jul 31 14:49:14.046596 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.046579    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e1cfecbe-4155-4b24-aad5-2277fd0c350e-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-m4hkx\" (UID: \"e1cfecbe-4155-4b24-aad5-2277fd0c350e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m4hkx"
Jul 31 14:49:14.095231 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.095201    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5knbp\" (UniqueName: \"kubernetes.io/projected/e1cfecbe-4155-4b24-aad5-2277fd0c350e-kube-api-access-5knbp\") pod \"cluster-monitoring-operator-6f55c4f58d-m4hkx\" (UID: \"e1cfecbe-4155-4b24-aad5-2277fd0c350e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m4hkx"
Jul 31 14:49:14.146615 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.146580    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzlfm\" (UniqueName: \"kubernetes.io/projected/473e45a2-a84a-4a9f-8950-959e755baefa-kube-api-access-jzlfm\") pod \"service-ca-operator-784f97bcf6-mngjp\" (UID: \"473e45a2-a84a-4a9f-8950-959e755baefa\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mngjp"
Jul 31 14:49:14.146615 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.146613    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1380fa66-45b6-48b5-aee6-18f70ca49ebf-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-c6dsw\" (UID: \"1380fa66-45b6-48b5-aee6-18f70ca49ebf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-c6dsw"
Jul 31 14:49:14.146907 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.146642    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/473e45a2-a84a-4a9f-8950-959e755baefa-config\") pod \"service-ca-operator-784f97bcf6-mngjp\" (UID: \"473e45a2-a84a-4a9f-8950-959e755baefa\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mngjp"
Jul 31 14:49:14.146907 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.146660    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wsn7\" (UniqueName: \"kubernetes.io/projected/1380fa66-45b6-48b5-aee6-18f70ca49ebf-kube-api-access-9wsn7\") pod \"kube-storage-version-migrator-operator-7878f4b88c-c6dsw\" (UID: \"1380fa66-45b6-48b5-aee6-18f70ca49ebf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-c6dsw"
Jul 31 14:49:14.146907 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.146680    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/28740bd0-9d46-4230-b1c7-7364aafa2066-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-s787l\" (UID: \"28740bd0-9d46-4230-b1c7-7364aafa2066\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s787l"
Jul 31 14:49:14.146907 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.146752    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcdlm\" (UniqueName: \"kubernetes.io/projected/689c6bab-fc78-4cc4-81be-50463036b079-kube-api-access-fcdlm\") pod \"volume-data-source-validator-5b8776669-rgrwh\" (UID: \"689c6bab-fc78-4cc4-81be-50463036b079\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-rgrwh"
Jul 31 14:49:14.146907 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.146794    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/473e45a2-a84a-4a9f-8950-959e755baefa-serving-cert\") pod \"service-ca-operator-784f97bcf6-mngjp\" (UID: \"473e45a2-a84a-4a9f-8950-959e755baefa\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mngjp"
Jul 31 14:49:14.146907 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.146830    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1380fa66-45b6-48b5-aee6-18f70ca49ebf-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-c6dsw\" (UID: \"1380fa66-45b6-48b5-aee6-18f70ca49ebf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-c6dsw"
Jul 31 14:49:14.146907 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.146856    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h2c7\" (UniqueName: \"kubernetes.io/projected/28740bd0-9d46-4230-b1c7-7364aafa2066-kube-api-access-9h2c7\") pod \"cluster-samples-operator-559cbf5745-s787l\" (UID: \"28740bd0-9d46-4230-b1c7-7364aafa2066\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s787l"
Jul 31 14:49:14.147254 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.147238    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/473e45a2-a84a-4a9f-8950-959e755baefa-config\") pod \"service-ca-operator-784f97bcf6-mngjp\" (UID: \"473e45a2-a84a-4a9f-8950-959e755baefa\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mngjp"
Jul 31 14:49:14.149161 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.149141    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/473e45a2-a84a-4a9f-8950-959e755baefa-serving-cert\") pod \"service-ca-operator-784f97bcf6-mngjp\" (UID: \"473e45a2-a84a-4a9f-8950-959e755baefa\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mngjp"
Jul 31 14:49:14.152887 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.152778    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1380fa66-45b6-48b5-aee6-18f70ca49ebf-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-c6dsw\" (UID: \"1380fa66-45b6-48b5-aee6-18f70ca49ebf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-c6dsw"
Jul 31 14:49:14.154374 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.154350    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1380fa66-45b6-48b5-aee6-18f70ca49ebf-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-c6dsw\" (UID: \"1380fa66-45b6-48b5-aee6-18f70ca49ebf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-c6dsw"
Jul 31 14:49:14.168350 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.168324    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcdlm\" (UniqueName: \"kubernetes.io/projected/689c6bab-fc78-4cc4-81be-50463036b079-kube-api-access-fcdlm\") pod \"volume-data-source-validator-5b8776669-rgrwh\" (UID: \"689c6bab-fc78-4cc4-81be-50463036b079\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-rgrwh"
Jul 31 14:49:14.169089 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.169069    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wsn7\" (UniqueName: \"kubernetes.io/projected/1380fa66-45b6-48b5-aee6-18f70ca49ebf-kube-api-access-9wsn7\") pod \"kube-storage-version-migrator-operator-7878f4b88c-c6dsw\" (UID: \"1380fa66-45b6-48b5-aee6-18f70ca49ebf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-c6dsw"
Jul 31 14:49:14.177525 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.177501    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzlfm\" (UniqueName: \"kubernetes.io/projected/473e45a2-a84a-4a9f-8950-959e755baefa-kube-api-access-jzlfm\") pod \"service-ca-operator-784f97bcf6-mngjp\" (UID: \"473e45a2-a84a-4a9f-8950-959e755baefa\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mngjp"
Jul 31 14:49:14.234115 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.234081    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-c6dsw"
Jul 31 14:49:14.247497 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.247461    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h2c7\" (UniqueName: \"kubernetes.io/projected/28740bd0-9d46-4230-b1c7-7364aafa2066-kube-api-access-9h2c7\") pod \"cluster-samples-operator-559cbf5745-s787l\" (UID: \"28740bd0-9d46-4230-b1c7-7364aafa2066\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s787l"
Jul 31 14:49:14.247652 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.247559    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/28740bd0-9d46-4230-b1c7-7364aafa2066-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-s787l\" (UID: \"28740bd0-9d46-4230-b1c7-7364aafa2066\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s787l"
Jul 31 14:49:14.247718 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:14.247681    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 31 14:49:14.247808 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:14.247773    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28740bd0-9d46-4230-b1c7-7364aafa2066-samples-operator-tls podName:28740bd0-9d46-4230-b1c7-7364aafa2066 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:14.747753151 +0000 UTC m=+51.059310752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/28740bd0-9d46-4230-b1c7-7364aafa2066-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-s787l" (UID: "28740bd0-9d46-4230-b1c7-7364aafa2066") : secret "samples-operator-tls" not found
Jul 31 14:49:14.264752 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.264709    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-rgrwh"
Jul 31 14:49:14.264752 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.264724    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mngjp"
Jul 31 14:49:14.274957 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.274922    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h2c7\" (UniqueName: \"kubernetes.io/projected/28740bd0-9d46-4230-b1c7-7364aafa2066-kube-api-access-9h2c7\") pod \"cluster-samples-operator-559cbf5745-s787l\" (UID: \"28740bd0-9d46-4230-b1c7-7364aafa2066\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s787l"
Jul 31 14:49:14.430817 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.430786    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-c6dsw"]
Jul 31 14:49:14.434776 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:49:14.434751    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1380fa66_45b6_48b5_aee6_18f70ca49ebf.slice/crio-295f5b2e312b70d89f5ba9b5043eb1da1aeccb16350188704e15e7ca4a2d0b86 WatchSource:0}: Error finding container 295f5b2e312b70d89f5ba9b5043eb1da1aeccb16350188704e15e7ca4a2d0b86: Status 404 returned error can't find the container with id 295f5b2e312b70d89f5ba9b5043eb1da1aeccb16350188704e15e7ca4a2d0b86
Jul 31 14:49:14.447344 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.447321    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-rgrwh"]
Jul 31 14:49:14.450647 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:49:14.450622    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod689c6bab_fc78_4cc4_81be_50463036b079.slice/crio-ad6d8af7fd1b3039ea33ed0164aef25b99a887a206072bc66bd8a4f94341ee8a WatchSource:0}: Error finding container ad6d8af7fd1b3039ea33ed0164aef25b99a887a206072bc66bd8a4f94341ee8a: Status 404 returned error can't find the container with id ad6d8af7fd1b3039ea33ed0164aef25b99a887a206072bc66bd8a4f94341ee8a
Jul 31 14:49:14.460519 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.460439    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-mngjp"]
Jul 31 14:49:14.552059 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.551964    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e1cfecbe-4155-4b24-aad5-2277fd0c350e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-m4hkx\" (UID: \"e1cfecbe-4155-4b24-aad5-2277fd0c350e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m4hkx"
Jul 31 14:49:14.552206 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:14.552129    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 14:49:14.552245 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:14.552208    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1cfecbe-4155-4b24-aad5-2277fd0c350e-cluster-monitoring-operator-tls podName:e1cfecbe-4155-4b24-aad5-2277fd0c350e nodeName:}" failed. No retries permitted until 2026-07-31 14:49:15.552193009 +0000 UTC m=+51.863750609 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e1cfecbe-4155-4b24-aad5-2277fd0c350e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-m4hkx" (UID: "e1cfecbe-4155-4b24-aad5-2277fd0c350e") : secret "cluster-monitoring-operator-tls" not found
Jul 31 14:49:14.753027 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:14.752995    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/28740bd0-9d46-4230-b1c7-7364aafa2066-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-s787l\" (UID: \"28740bd0-9d46-4230-b1c7-7364aafa2066\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s787l"
Jul 31 14:49:14.753182 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:14.753108    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 31 14:49:14.753182 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:14.753158    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28740bd0-9d46-4230-b1c7-7364aafa2066-samples-operator-tls podName:28740bd0-9d46-4230-b1c7-7364aafa2066 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:15.753143488 +0000 UTC m=+52.064701088 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/28740bd0-9d46-4230-b1c7-7364aafa2066-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-s787l" (UID: "28740bd0-9d46-4230-b1c7-7364aafa2066") : secret "samples-operator-tls" not found
Jul 31 14:49:15.362287 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:15.362154    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee553c57-d835-4598-a0ee-b6fe7d685490-metrics-tls\") pod \"dns-default-2mr24\" (UID: \"ee553c57-d835-4598-a0ee-b6fe7d685490\") " pod="openshift-dns/dns-default-2mr24"
Jul 31 14:49:15.362784 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:15.362334    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:49:15.362784 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:15.362410    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee553c57-d835-4598-a0ee-b6fe7d685490-metrics-tls podName:ee553c57-d835-4598-a0ee-b6fe7d685490 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:31.362386391 +0000 UTC m=+67.673943994 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ee553c57-d835-4598-a0ee-b6fe7d685490-metrics-tls") pod "dns-default-2mr24" (UID: "ee553c57-d835-4598-a0ee-b6fe7d685490") : secret "dns-default-metrics-tls" not found
Jul 31 14:49:15.419723 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:15.419683    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mngjp" event={"ID":"473e45a2-a84a-4a9f-8950-959e755baefa","Type":"ContainerStarted","Data":"e82afdd9ce743576fcdafc9c008aecf413ea9ea2e530275840f9cb5f258d8de2"}
Jul 31 14:49:15.421296 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:15.421269    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-rgrwh" event={"ID":"689c6bab-fc78-4cc4-81be-50463036b079","Type":"ContainerStarted","Data":"ad6d8af7fd1b3039ea33ed0164aef25b99a887a206072bc66bd8a4f94341ee8a"}
Jul 31 14:49:15.423266 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:15.423237    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-c6dsw" event={"ID":"1380fa66-45b6-48b5-aee6-18f70ca49ebf","Type":"ContainerStarted","Data":"295f5b2e312b70d89f5ba9b5043eb1da1aeccb16350188704e15e7ca4a2d0b86"}
Jul 31 14:49:15.564669 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:15.564581    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e1cfecbe-4155-4b24-aad5-2277fd0c350e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-m4hkx\" (UID: \"e1cfecbe-4155-4b24-aad5-2277fd0c350e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m4hkx"
Jul 31 14:49:15.564878 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:15.564767    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 14:49:15.564878 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:15.564847    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1cfecbe-4155-4b24-aad5-2277fd0c350e-cluster-monitoring-operator-tls podName:e1cfecbe-4155-4b24-aad5-2277fd0c350e nodeName:}" failed. No retries permitted until 2026-07-31 14:49:17.564827622 +0000 UTC m=+53.876385226 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e1cfecbe-4155-4b24-aad5-2277fd0c350e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-m4hkx" (UID: "e1cfecbe-4155-4b24-aad5-2277fd0c350e") : secret "cluster-monitoring-operator-tls" not found
Jul 31 14:49:15.765959 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:15.765873    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/28740bd0-9d46-4230-b1c7-7364aafa2066-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-s787l\" (UID: \"28740bd0-9d46-4230-b1c7-7364aafa2066\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s787l"
Jul 31 14:49:15.766126 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:15.766042    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 31 14:49:15.766126 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:15.766116    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28740bd0-9d46-4230-b1c7-7364aafa2066-samples-operator-tls podName:28740bd0-9d46-4230-b1c7-7364aafa2066 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:17.766097292 +0000 UTC m=+54.077654915 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/28740bd0-9d46-4230-b1c7-7364aafa2066-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-s787l" (UID: "28740bd0-9d46-4230-b1c7-7364aafa2066") : secret "samples-operator-tls" not found
Jul 31 14:49:17.581679 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:17.581648    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e1cfecbe-4155-4b24-aad5-2277fd0c350e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-m4hkx\" (UID: \"e1cfecbe-4155-4b24-aad5-2277fd0c350e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m4hkx"
Jul 31 14:49:17.582008 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:17.581876    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 14:49:17.582008 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:17.581946    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1cfecbe-4155-4b24-aad5-2277fd0c350e-cluster-monitoring-operator-tls podName:e1cfecbe-4155-4b24-aad5-2277fd0c350e nodeName:}" failed. No retries permitted until 2026-07-31 14:49:21.581929006 +0000 UTC m=+57.893486608 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e1cfecbe-4155-4b24-aad5-2277fd0c350e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-m4hkx" (UID: "e1cfecbe-4155-4b24-aad5-2277fd0c350e") : secret "cluster-monitoring-operator-tls" not found
Jul 31 14:49:17.783014 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:17.782979    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/28740bd0-9d46-4230-b1c7-7364aafa2066-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-s787l\" (UID: \"28740bd0-9d46-4230-b1c7-7364aafa2066\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s787l"
Jul 31 14:49:17.783184 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:17.783161    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 31 14:49:17.783256 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:17.783246    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28740bd0-9d46-4230-b1c7-7364aafa2066-samples-operator-tls podName:28740bd0-9d46-4230-b1c7-7364aafa2066 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:21.783224363 +0000 UTC m=+58.094781970 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/28740bd0-9d46-4230-b1c7-7364aafa2066-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-s787l" (UID: "28740bd0-9d46-4230-b1c7-7364aafa2066") : secret "samples-operator-tls" not found
Jul 31 14:49:18.431696 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:18.431662    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mngjp" event={"ID":"473e45a2-a84a-4a9f-8950-959e755baefa","Type":"ContainerStarted","Data":"44b805b8a24605b9f3b4714a8246313331e4f52d0b67f7f3071657ea20c31bfe"}
Jul 31 14:49:18.433143 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:18.433117    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-rgrwh" event={"ID":"689c6bab-fc78-4cc4-81be-50463036b079","Type":"ContainerStarted","Data":"f1fc892e72172bfe85a17bbfdca94ac8e4fe9cfc6ffb25fb177d76b22ff282c2"}
Jul 31 14:49:18.434550 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:18.434524    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-c6dsw" event={"ID":"1380fa66-45b6-48b5-aee6-18f70ca49ebf","Type":"ContainerStarted","Data":"5936c893de825417ab34b9f5d62dfbc46b3a56ee200df026f81b0fd49738b08f"}
Jul 31 14:49:18.455791 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:18.455721    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mngjp" podStartSLOduration=2.392979801 podStartE2EDuration="5.455704249s" totalImagesPullingTime="3.062724448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:18.453904615 +0000 UTC m=+54.765462237" watchObservedRunningTime="2026-07-31 14:49:18.455704249 +0000 UTC m=+54.767261849"
Jul 31 14:49:18.486474 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:18.485904    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-rgrwh" podStartSLOduration=2.416287614 podStartE2EDuration="5.485880055s" totalImagesPullingTime="3.069592441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:18.480869486 +0000 UTC m=+54.792427132" watchObservedRunningTime="2026-07-31 14:49:18.485880055 +0000 UTC m=+54.797437678"
Jul 31 14:49:18.520440 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:18.520377    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-c6dsw" podStartSLOduration=2.435243354 podStartE2EDuration="5.520358865s" totalImagesPullingTime="3.085115511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:18.519922184 +0000 UTC m=+54.831479811" watchObservedRunningTime="2026-07-31 14:49:18.520358865 +0000 UTC m=+54.831916486"
Jul 31 14:49:20.058105 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:20.058071    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dnt6w"]
Jul 31 14:49:20.060924 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:20.060906    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dnt6w"
Jul 31 14:49:20.064439 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:20.064418    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 31 14:49:20.064664 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:20.064648    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j66bv\""
Jul 31 14:49:20.065436 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:20.065422    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 31 14:49:20.066146 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:20.066124    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 31 14:49:20.066840 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:20.066817    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 31 14:49:20.077442 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:20.077413    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dnt6w"]
Jul 31 14:49:20.103604 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:20.103572    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/91c153e9-2281-4901-a232-1dcd8326f94d-data-volume\") pod \"insights-runtime-extractor-dnt6w\" (UID: \"91c153e9-2281-4901-a232-1dcd8326f94d\") " pod="openshift-insights/insights-runtime-extractor-dnt6w"
Jul 31 14:49:20.103604 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:20.103608    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/91c153e9-2281-4901-a232-1dcd8326f94d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnt6w\" (UID: \"91c153e9-2281-4901-a232-1dcd8326f94d\") " pod="openshift-insights/insights-runtime-extractor-dnt6w"
Jul 31 14:49:20.103833 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:20.103667    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/91c153e9-2281-4901-a232-1dcd8326f94d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnt6w\" (UID: \"91c153e9-2281-4901-a232-1dcd8326f94d\") " pod="openshift-insights/insights-runtime-extractor-dnt6w"
Jul 31 14:49:20.103833 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:20.103791    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/91c153e9-2281-4901-a232-1dcd8326f94d-crio-socket\") pod \"insights-runtime-extractor-dnt6w\" (UID: \"91c153e9-2281-4901-a232-1dcd8326f94d\") " pod="openshift-insights/insights-runtime-extractor-dnt6w"
Jul 31 14:49:20.103833 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:20.103816    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8cnl\" (UniqueName: \"kubernetes.io/projected/91c153e9-2281-4901-a232-1dcd8326f94d-kube-api-access-s8cnl\") pod \"insights-runtime-extractor-dnt6w\" (UID: \"91c153e9-2281-4901-a232-1dcd8326f94d\") " pod="openshift-insights/insights-runtime-extractor-dnt6w"
Jul 31 14:49:20.204679 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:20.204637    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/91c153e9-2281-4901-a232-1dcd8326f94d-data-volume\") pod \"insights-runtime-extractor-dnt6w\" (UID: \"91c153e9-2281-4901-a232-1dcd8326f94d\") " pod="openshift-insights/insights-runtime-extractor-dnt6w"
Jul 31 14:49:20.204679 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:20.204674    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/91c153e9-2281-4901-a232-1dcd8326f94d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnt6w\" (UID: \"91c153e9-2281-4901-a232-1dcd8326f94d\") " pod="openshift-insights/insights-runtime-extractor-dnt6w"
Jul 31 14:49:20.204921 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:20.204714    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/91c153e9-2281-4901-a232-1dcd8326f94d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnt6w\" (UID: \"91c153e9-2281-4901-a232-1dcd8326f94d\") " pod="openshift-insights/insights-runtime-extractor-dnt6w"
Jul 31 14:49:20.204921 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:20.204785    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/91c153e9-2281-4901-a232-1dcd8326f94d-crio-socket\") pod \"insights-runtime-extractor-dnt6w\" (UID: \"91c153e9-2281-4901-a232-1dcd8326f94d\") " pod="openshift-insights/insights-runtime-extractor-dnt6w"
Jul 31 14:49:20.204921 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:20.204803    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8cnl\" (UniqueName: \"kubernetes.io/projected/91c153e9-2281-4901-a232-1dcd8326f94d-kube-api-access-s8cnl\") pod \"insights-runtime-extractor-dnt6w\" (UID: \"91c153e9-2281-4901-a232-1dcd8326f94d\") " pod="openshift-insights/insights-runtime-extractor-dnt6w"
Jul 31 14:49:20.204921 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:20.204860    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:49:20.204921 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:20.204873    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:20.204921 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:20.204868    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/91c153e9-2281-4901-a232-1dcd8326f94d-crio-socket\") pod \"insights-runtime-extractor-dnt6w\" (UID: \"91c153e9-2281-4901-a232-1dcd8326f94d\") " pod="openshift-insights/insights-runtime-extractor-dnt6w"
Jul 31 14:49:20.205127 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:20.204935    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91c153e9-2281-4901-a232-1dcd8326f94d-insights-runtime-extractor-tls podName:91c153e9-2281-4901-a232-1dcd8326f94d nodeName:}" failed. No retries permitted until 2026-07-31 14:49:20.704914688 +0000 UTC m=+57.016472290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/91c153e9-2281-4901-a232-1dcd8326f94d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dnt6w" (UID: "91c153e9-2281-4901-a232-1dcd8326f94d") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:20.205127 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:20.204957    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91c153e9-2281-4901-a232-1dcd8326f94d-kube-rbac-proxy-cm podName:91c153e9-2281-4901-a232-1dcd8326f94d nodeName:}" failed. No retries permitted until 2026-07-31 14:49:20.704944002 +0000 UTC m=+57.016501605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/91c153e9-2281-4901-a232-1dcd8326f94d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dnt6w" (UID: "91c153e9-2281-4901-a232-1dcd8326f94d") : configmap "kube-rbac-proxy" not found
Jul 31 14:49:20.205127 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:20.205002    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/91c153e9-2281-4901-a232-1dcd8326f94d-data-volume\") pod \"insights-runtime-extractor-dnt6w\" (UID: \"91c153e9-2281-4901-a232-1dcd8326f94d\") " pod="openshift-insights/insights-runtime-extractor-dnt6w"
Jul 31 14:49:20.230624 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:20.230586    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8cnl\" (UniqueName: \"kubernetes.io/projected/91c153e9-2281-4901-a232-1dcd8326f94d-kube-api-access-s8cnl\") pod \"insights-runtime-extractor-dnt6w\" (UID: \"91c153e9-2281-4901-a232-1dcd8326f94d\") " pod="openshift-insights/insights-runtime-extractor-dnt6w"
Jul 31 14:49:20.708315 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:20.708276    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/91c153e9-2281-4901-a232-1dcd8326f94d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnt6w\" (UID: \"91c153e9-2281-4901-a232-1dcd8326f94d\") " pod="openshift-insights/insights-runtime-extractor-dnt6w"
Jul 31 14:49:20.708496 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:20.708332    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/91c153e9-2281-4901-a232-1dcd8326f94d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnt6w\" (UID: \"91c153e9-2281-4901-a232-1dcd8326f94d\") " pod="openshift-insights/insights-runtime-extractor-dnt6w"
Jul 31 14:49:20.708496 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:20.708422    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:20.708496 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:20.708441    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:49:20.708496 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:20.708494    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91c153e9-2281-4901-a232-1dcd8326f94d-insights-runtime-extractor-tls podName:91c153e9-2281-4901-a232-1dcd8326f94d nodeName:}" failed. No retries permitted until 2026-07-31 14:49:21.708479459 +0000 UTC m=+58.020037059 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/91c153e9-2281-4901-a232-1dcd8326f94d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dnt6w" (UID: "91c153e9-2281-4901-a232-1dcd8326f94d") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:20.708632 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:20.708508    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91c153e9-2281-4901-a232-1dcd8326f94d-kube-rbac-proxy-cm podName:91c153e9-2281-4901-a232-1dcd8326f94d nodeName:}" failed. No retries permitted until 2026-07-31 14:49:21.708501869 +0000 UTC m=+58.020059469 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/91c153e9-2281-4901-a232-1dcd8326f94d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dnt6w" (UID: "91c153e9-2281-4901-a232-1dcd8326f94d") : configmap "kube-rbac-proxy" not found
Jul 31 14:49:21.614473 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:21.614435    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e1cfecbe-4155-4b24-aad5-2277fd0c350e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-m4hkx\" (UID: \"e1cfecbe-4155-4b24-aad5-2277fd0c350e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m4hkx"
Jul 31 14:49:21.614898 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:21.614568    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 14:49:21.614898 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:21.614633    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1cfecbe-4155-4b24-aad5-2277fd0c350e-cluster-monitoring-operator-tls podName:e1cfecbe-4155-4b24-aad5-2277fd0c350e nodeName:}" failed. No retries permitted until 2026-07-31 14:49:29.614618594 +0000 UTC m=+65.926176194 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e1cfecbe-4155-4b24-aad5-2277fd0c350e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-m4hkx" (UID: "e1cfecbe-4155-4b24-aad5-2277fd0c350e") : secret "cluster-monitoring-operator-tls" not found
Jul 31 14:49:21.715685 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:21.715652    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/91c153e9-2281-4901-a232-1dcd8326f94d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnt6w\" (UID: \"91c153e9-2281-4901-a232-1dcd8326f94d\") " pod="openshift-insights/insights-runtime-extractor-dnt6w"
Jul 31 14:49:21.715897 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:21.715802    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/91c153e9-2281-4901-a232-1dcd8326f94d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnt6w\" (UID: \"91c153e9-2281-4901-a232-1dcd8326f94d\") " pod="openshift-insights/insights-runtime-extractor-dnt6w"
Jul 31 14:49:21.715897 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:21.715805    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:49:21.716011 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:21.715900    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91c153e9-2281-4901-a232-1dcd8326f94d-kube-rbac-proxy-cm podName:91c153e9-2281-4901-a232-1dcd8326f94d nodeName:}" failed. No retries permitted until 2026-07-31 14:49:23.715878276 +0000 UTC m=+60.027435880 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/91c153e9-2281-4901-a232-1dcd8326f94d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dnt6w" (UID: "91c153e9-2281-4901-a232-1dcd8326f94d") : configmap "kube-rbac-proxy" not found
Jul 31 14:49:21.716011 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:21.715930    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:21.716011 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:21.715994    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91c153e9-2281-4901-a232-1dcd8326f94d-insights-runtime-extractor-tls podName:91c153e9-2281-4901-a232-1dcd8326f94d nodeName:}" failed. No retries permitted until 2026-07-31 14:49:23.715981286 +0000 UTC m=+60.027538887 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/91c153e9-2281-4901-a232-1dcd8326f94d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dnt6w" (UID: "91c153e9-2281-4901-a232-1dcd8326f94d") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:21.816817 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:21.816789    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/28740bd0-9d46-4230-b1c7-7364aafa2066-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-s787l\" (UID: \"28740bd0-9d46-4230-b1c7-7364aafa2066\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s787l"
Jul 31 14:49:21.816980 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:21.816912    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 31 14:49:21.816980 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:21.816973    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28740bd0-9d46-4230-b1c7-7364aafa2066-samples-operator-tls podName:28740bd0-9d46-4230-b1c7-7364aafa2066 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:29.816959071 +0000 UTC m=+66.128516675 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/28740bd0-9d46-4230-b1c7-7364aafa2066-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-s787l" (UID: "28740bd0-9d46-4230-b1c7-7364aafa2066") : secret "samples-operator-tls" not found
Jul 31 14:49:22.479332 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:22.479301    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mpkxl_54adc061-d1ce-4ba7-83cf-3b2968693d52/dns-node-resolver/0.log"
Jul 31 14:49:23.473178 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:23.473152    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bwcjx_16b5669e-0dde-49c0-854b-495854e99a82/node-ca/0.log"
Jul 31 14:49:23.734981 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:23.734905    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/91c153e9-2281-4901-a232-1dcd8326f94d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnt6w\" (UID: \"91c153e9-2281-4901-a232-1dcd8326f94d\") " pod="openshift-insights/insights-runtime-extractor-dnt6w"
Jul 31 14:49:23.734981 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:23.734959    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/91c153e9-2281-4901-a232-1dcd8326f94d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnt6w\" (UID: \"91c153e9-2281-4901-a232-1dcd8326f94d\") " pod="openshift-insights/insights-runtime-extractor-dnt6w"
Jul 31 14:49:23.735159 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:23.735050    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:49:23.735159 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:23.735050    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:23.735159 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:23.735107    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91c153e9-2281-4901-a232-1dcd8326f94d-insights-runtime-extractor-tls podName:91c153e9-2281-4901-a232-1dcd8326f94d nodeName:}" failed. No retries permitted until 2026-07-31 14:49:27.735091015 +0000 UTC m=+64.046648615 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/91c153e9-2281-4901-a232-1dcd8326f94d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dnt6w" (UID: "91c153e9-2281-4901-a232-1dcd8326f94d") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:23.735159 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:23.735123    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91c153e9-2281-4901-a232-1dcd8326f94d-kube-rbac-proxy-cm podName:91c153e9-2281-4901-a232-1dcd8326f94d nodeName:}" failed. No retries permitted until 2026-07-31 14:49:27.735116322 +0000 UTC m=+64.046673923 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/91c153e9-2281-4901-a232-1dcd8326f94d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dnt6w" (UID: "91c153e9-2281-4901-a232-1dcd8326f94d") : configmap "kube-rbac-proxy" not found
Jul 31 14:49:24.273979 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:24.273953    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fz26d_ba20a352-de91-49e8-a2ac-b5e1b9c37ae4/serve-healthcheck-canary/0.log"
Jul 31 14:49:25.274054 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:25.274018    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-c6dsw_1380fa66-45b6-48b5-aee6-18f70ca49ebf/kube-storage-version-migrator-operator/0.log"
Jul 31 14:49:26.369913 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:26.369877    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qp8cd"
Jul 31 14:49:27.766692 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:27.766637    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/91c153e9-2281-4901-a232-1dcd8326f94d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnt6w\" (UID: \"91c153e9-2281-4901-a232-1dcd8326f94d\") " pod="openshift-insights/insights-runtime-extractor-dnt6w"
Jul 31 14:49:27.767106 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:27.766710    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/91c153e9-2281-4901-a232-1dcd8326f94d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnt6w\" (UID: \"91c153e9-2281-4901-a232-1dcd8326f94d\") " pod="openshift-insights/insights-runtime-extractor-dnt6w"
Jul 31 14:49:27.767106 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:27.766795    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:49:27.767106 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:27.766826    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:27.767106 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:27.766876    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91c153e9-2281-4901-a232-1dcd8326f94d-insights-runtime-extractor-tls podName:91c153e9-2281-4901-a232-1dcd8326f94d nodeName:}" failed. No retries permitted until 2026-07-31 14:49:35.766856767 +0000 UTC m=+72.078414383 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/91c153e9-2281-4901-a232-1dcd8326f94d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dnt6w" (UID: "91c153e9-2281-4901-a232-1dcd8326f94d") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:27.767106 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:27.766894    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91c153e9-2281-4901-a232-1dcd8326f94d-kube-rbac-proxy-cm podName:91c153e9-2281-4901-a232-1dcd8326f94d nodeName:}" failed. No retries permitted until 2026-07-31 14:49:35.766884679 +0000 UTC m=+72.078442309 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/91c153e9-2281-4901-a232-1dcd8326f94d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dnt6w" (UID: "91c153e9-2281-4901-a232-1dcd8326f94d") : configmap "kube-rbac-proxy" not found
Jul 31 14:49:29.679995 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:29.679952    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e1cfecbe-4155-4b24-aad5-2277fd0c350e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-m4hkx\" (UID: \"e1cfecbe-4155-4b24-aad5-2277fd0c350e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m4hkx"
Jul 31 14:49:29.680384 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:29.680109    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 14:49:29.680384 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:29.680181    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1cfecbe-4155-4b24-aad5-2277fd0c350e-cluster-monitoring-operator-tls podName:e1cfecbe-4155-4b24-aad5-2277fd0c350e nodeName:}" failed. No retries permitted until 2026-07-31 14:49:45.680163209 +0000 UTC m=+81.991720809 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e1cfecbe-4155-4b24-aad5-2277fd0c350e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-m4hkx" (UID: "e1cfecbe-4155-4b24-aad5-2277fd0c350e") : secret "cluster-monitoring-operator-tls" not found
Jul 31 14:49:29.881822 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:29.881785    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355d9cbb-1867-49cc-8ec2-564b4aa6daea-metrics-certs\") pod \"network-metrics-daemon-b45r4\" (UID: \"355d9cbb-1867-49cc-8ec2-564b4aa6daea\") " pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:49:29.881822 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:29.881827    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/28740bd0-9d46-4230-b1c7-7364aafa2066-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-s787l\" (UID: \"28740bd0-9d46-4230-b1c7-7364aafa2066\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s787l"
Jul 31 14:49:29.884167 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:29.884145    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 14:49:29.884472 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:29.884447    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/28740bd0-9d46-4230-b1c7-7364aafa2066-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-s787l\" (UID: \"28740bd0-9d46-4230-b1c7-7364aafa2066\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s787l"
Jul 31 14:49:29.892233 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:29.892210    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 31 14:49:29.892355 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:49:29.892269    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/355d9cbb-1867-49cc-8ec2-564b4aa6daea-metrics-certs podName:355d9cbb-1867-49cc-8ec2-564b4aa6daea nodeName:}" failed. No retries permitted until 2026-07-31 14:50:33.892253639 +0000 UTC m=+130.203811238 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/355d9cbb-1867-49cc-8ec2-564b4aa6daea-metrics-certs") pod "network-metrics-daemon-b45r4" (UID: "355d9cbb-1867-49cc-8ec2-564b4aa6daea") : secret "metrics-daemon-secret" not found
Jul 31 14:49:29.893519 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:29.893503    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-w52wl\""
Jul 31 14:49:29.901816 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:29.901797    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s787l"
Jul 31 14:49:29.983236 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:29.983206    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d947335a-f9d7-4fcd-b9e1-457477ab7f72-original-pull-secret\") pod \"global-pull-secret-syncer-nk8pw\" (UID: \"d947335a-f9d7-4fcd-b9e1-457477ab7f72\") " pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:49:29.992050 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:29.992022    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 14:49:29.996887 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:29.996859    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d947335a-f9d7-4fcd-b9e1-457477ab7f72-original-pull-secret\") pod \"global-pull-secret-syncer-nk8pw\" (UID: \"d947335a-f9d7-4fcd-b9e1-457477ab7f72\") " pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:49:30.073781 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:30.073756    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s787l"]
Jul 31 14:49:30.083846 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:30.083815    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97xf4\" (UniqueName: \"kubernetes.io/projected/567a8972-6aaf-499f-a8d8-a83da117da60-kube-api-access-97xf4\") pod \"network-check-target-6vfsr\" (UID: \"567a8972-6aaf-499f-a8d8-a83da117da60\") " pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:49:30.094911 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:30.094889    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 14:49:30.111612 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:30.111591    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 14:49:30.118241 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:30.117640    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97xf4\" (UniqueName: \"kubernetes.io/projected/567a8972-6aaf-499f-a8d8-a83da117da60-kube-api-access-97xf4\") pod \"network-check-target-6vfsr\" (UID: \"567a8972-6aaf-499f-a8d8-a83da117da60\") " pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:49:30.227496 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:30.227392    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nk8pw"
Jul 31 14:49:30.331581 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:30.331556    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wtnft\""
Jul 31 14:49:30.338307 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:30.338288    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:49:30.363560 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:30.363526    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nk8pw"]
Jul 31 14:49:30.367718 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:49:30.367692    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd947335a_f9d7_4fcd_b9e1_457477ab7f72.slice/crio-6d633fac8c1b42134f89cce32d7ac67b693c9fecde379179ee4238a6bdd308c1 WatchSource:0}: Error finding container 6d633fac8c1b42134f89cce32d7ac67b693c9fecde379179ee4238a6bdd308c1: Status 404 returned error can't find the container with id 6d633fac8c1b42134f89cce32d7ac67b693c9fecde379179ee4238a6bdd308c1
Jul 31 14:49:30.464893 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:30.464857    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s787l" event={"ID":"28740bd0-9d46-4230-b1c7-7364aafa2066","Type":"ContainerStarted","Data":"f91cb7f4039393e4a2e48d89ea0ab12b8a1f15d42a25d2733299db0fb183f6fa"}
Jul 31 14:49:30.465881 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:30.465853    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nk8pw" event={"ID":"d947335a-f9d7-4fcd-b9e1-457477ab7f72","Type":"ContainerStarted","Data":"6d633fac8c1b42134f89cce32d7ac67b693c9fecde379179ee4238a6bdd308c1"}
Jul 31 14:49:30.467601 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:30.467576    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6vfsr"]
Jul 31 14:49:30.470822 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:49:30.470799    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567a8972_6aaf_499f_a8d8_a83da117da60.slice/crio-924ae1a5d9f8247458656f98cdb0dc248337c3bc216e489100baac6ec0940905 WatchSource:0}: Error finding container 924ae1a5d9f8247458656f98cdb0dc248337c3bc216e489100baac6ec0940905: Status 404 returned error can't find the container with id 924ae1a5d9f8247458656f98cdb0dc248337c3bc216e489100baac6ec0940905
Jul 31 14:49:31.398443 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:31.397816    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee553c57-d835-4598-a0ee-b6fe7d685490-metrics-tls\") pod \"dns-default-2mr24\" (UID: \"ee553c57-d835-4598-a0ee-b6fe7d685490\") " pod="openshift-dns/dns-default-2mr24"
Jul 31 14:49:31.401109 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:31.401076    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee553c57-d835-4598-a0ee-b6fe7d685490-metrics-tls\") pod \"dns-default-2mr24\" (UID: \"ee553c57-d835-4598-a0ee-b6fe7d685490\") " pod="openshift-dns/dns-default-2mr24"
Jul 31 14:49:31.471701 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:31.471662    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6vfsr" event={"ID":"567a8972-6aaf-499f-a8d8-a83da117da60","Type":"ContainerStarted","Data":"924ae1a5d9f8247458656f98cdb0dc248337c3bc216e489100baac6ec0940905"}
Jul 31 14:49:31.657997 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:31.657893    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-plcv4\""
Jul 31 14:49:31.666222 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:31.666193    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2mr24"
Jul 31 14:49:32.015120 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:32.015055    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2mr24"]
Jul 31 14:49:32.018862 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:49:32.018830    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee553c57_d835_4598_a0ee_b6fe7d685490.slice/crio-96ff1cc8c22e1ac66065bfacc92a00a2dde144f18ea06131bdb59df559ee29e1 WatchSource:0}: Error finding container 96ff1cc8c22e1ac66065bfacc92a00a2dde144f18ea06131bdb59df559ee29e1: Status 404 returned error can't find the container with id 96ff1cc8c22e1ac66065bfacc92a00a2dde144f18ea06131bdb59df559ee29e1
Jul 31 14:49:32.479145 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:32.479111    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s787l" event={"ID":"28740bd0-9d46-4230-b1c7-7364aafa2066","Type":"ContainerStarted","Data":"b92a2c80b79d61b6db5e87399812601854ad6a8bfd05334dcfee48203205fa53"}
Jul 31 14:49:32.479609 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:32.479234    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s787l" event={"ID":"28740bd0-9d46-4230-b1c7-7364aafa2066","Type":"ContainerStarted","Data":"9d455dc0c928addfb3a930366e83c435467c1e6e888f6d456f1971cd2bceea54"}
Jul 31 14:49:32.481238 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:32.481195    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2mr24" event={"ID":"ee553c57-d835-4598-a0ee-b6fe7d685490","Type":"ContainerStarted","Data":"96ff1cc8c22e1ac66065bfacc92a00a2dde144f18ea06131bdb59df559ee29e1"}
Jul 31 14:49:32.501515 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:32.501454    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-s787l" podStartSLOduration=17.429134556 podStartE2EDuration="19.501431921s" totalImagesPullingTime="2.072297365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:32.500882471 +0000 UTC m=+68.812440108" watchObservedRunningTime="2026-07-31 14:49:32.501431921 +0000 UTC m=+68.812989526"
Jul 31 14:49:35.839853 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:35.839811    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/91c153e9-2281-4901-a232-1dcd8326f94d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnt6w\" (UID: \"91c153e9-2281-4901-a232-1dcd8326f94d\") " pod="openshift-insights/insights-runtime-extractor-dnt6w"
Jul 31 14:49:35.840429 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:35.839892    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/91c153e9-2281-4901-a232-1dcd8326f94d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnt6w\" (UID: \"91c153e9-2281-4901-a232-1dcd8326f94d\") " pod="openshift-insights/insights-runtime-extractor-dnt6w"
Jul 31 14:49:35.840566 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:35.840540    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/91c153e9-2281-4901-a232-1dcd8326f94d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dnt6w\" (UID: \"91c153e9-2281-4901-a232-1dcd8326f94d\") " pod="openshift-insights/insights-runtime-extractor-dnt6w"
Jul 31 14:49:35.842633 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:35.842605    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/91c153e9-2281-4901-a232-1dcd8326f94d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dnt6w\" (UID: \"91c153e9-2281-4901-a232-1dcd8326f94d\") " pod="openshift-insights/insights-runtime-extractor-dnt6w"
Jul 31 14:49:35.974199 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:35.974177    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j66bv\""
Jul 31 14:49:35.980481 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:35.979488    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dnt6w"
Jul 31 14:49:36.160975 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:36.160946    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dnt6w"]
Jul 31 14:49:36.164445 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:49:36.164415    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c153e9_2281_4901_a232_1dcd8326f94d.slice/crio-bac07a86b7414d1005b084f6254d900fca49cd5235fd64c04e54788b6f187e4a WatchSource:0}: Error finding container bac07a86b7414d1005b084f6254d900fca49cd5235fd64c04e54788b6f187e4a: Status 404 returned error can't find the container with id bac07a86b7414d1005b084f6254d900fca49cd5235fd64c04e54788b6f187e4a
Jul 31 14:49:36.497553 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:36.497512    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6vfsr" event={"ID":"567a8972-6aaf-499f-a8d8-a83da117da60","Type":"ContainerStarted","Data":"03894a2fbf3c7014f3026a88a11bc8d3eae3d1cdf9c952c73c4e6b6484e928d0"}
Jul 31 14:49:36.497760 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:36.497614    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:49:36.499020 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:36.498990    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nk8pw" event={"ID":"d947335a-f9d7-4fcd-b9e1-457477ab7f72","Type":"ContainerStarted","Data":"87b34027b0eac04a3711ed08df02a5522ca479fe9641900b9b73b8fb6002ef4f"}
Jul 31 14:49:36.500556 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:36.500533    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2mr24" event={"ID":"ee553c57-d835-4598-a0ee-b6fe7d685490","Type":"ContainerStarted","Data":"a55f2ae0d2e0dcd76e3b08ae044e8c4a4067f6057bef23b183363add33cb9d2a"}
Jul 31 14:49:36.500674 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:36.500570    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2mr24" event={"ID":"ee553c57-d835-4598-a0ee-b6fe7d685490","Type":"ContainerStarted","Data":"db5063d84ea305a2bfbeae097e809813b20abf530bf3bf233558096b79bd52ec"}
Jul 31 14:49:36.500674 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:36.500642    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2mr24"
Jul 31 14:49:36.501949 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:36.501929    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dnt6w" event={"ID":"91c153e9-2281-4901-a232-1dcd8326f94d","Type":"ContainerStarted","Data":"e94709a4ec91121ad6a8ba5a7ec0096766ee9d2af8e978be6a0b5c62006d74d6"}
Jul 31 14:49:36.502044 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:36.501966    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dnt6w" event={"ID":"91c153e9-2281-4901-a232-1dcd8326f94d","Type":"ContainerStarted","Data":"bac07a86b7414d1005b084f6254d900fca49cd5235fd64c04e54788b6f187e4a"}
Jul 31 14:49:36.520093 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:36.520043    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6vfsr" podStartSLOduration=67.023439632 podStartE2EDuration="1m12.520029935s" totalImagesPullingTime="5.496590303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:36.519747992 +0000 UTC m=+72.831305607" watchObservedRunningTime="2026-07-31 14:49:36.520029935 +0000 UTC m=+72.831587535"
Jul 31 14:49:36.537119 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:36.537070    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nk8pw" podStartSLOduration=66.923644969 podStartE2EDuration="1m12.537054805s" totalImagesPullingTime="5.613409836s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:36.537032463 +0000 UTC m=+72.848590067" watchObservedRunningTime="2026-07-31 14:49:36.537054805 +0000 UTC m=+72.848612421"
Jul 31 14:49:36.556856 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:36.556806    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2mr24" podStartSLOduration=33.762394883 podStartE2EDuration="37.556789905s" totalImagesPullingTime="3.794395022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:36.556081359 +0000 UTC m=+72.867638974" watchObservedRunningTime="2026-07-31 14:49:36.556789905 +0000 UTC m=+72.868347528"
Jul 31 14:49:37.508723 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:37.508631    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dnt6w" event={"ID":"91c153e9-2281-4901-a232-1dcd8326f94d","Type":"ContainerStarted","Data":"f5df162cdd3ef03e2b622bc5e0226db3a80440eac83e90b0e6957b6f8aaf7c5a"}
Jul 31 14:49:39.520815 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:39.520782    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dnt6w" event={"ID":"91c153e9-2281-4901-a232-1dcd8326f94d","Type":"ContainerStarted","Data":"f5b3ff016b8a61b4220e558e0ac1f313cdfe46c38f52a8a25989f99bc0752330"}
Jul 31 14:49:39.541545 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:39.541495    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dnt6w" podStartSLOduration=17.363714242 podStartE2EDuration="19.541480445s" totalImagesPullingTime="2.177766203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:49:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:39.540696469 +0000 UTC m=+75.852254090" watchObservedRunningTime="2026-07-31 14:49:39.541480445 +0000 UTC m=+75.853038067"
Jul 31 14:49:43.869910 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:43.869877    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59d4fdc6b8-r7hdk"]
Jul 31 14:49:43.871789 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:43.871774    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59d4fdc6b8-r7hdk"
Jul 31 14:49:43.879445 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:43.879420    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 31 14:49:43.879794 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:43.879765    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 31 14:49:43.879954 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:43.879894    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-795zg\""
Jul 31 14:49:43.880062 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:43.880001    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb9599594-9n865"]
Jul 31 14:49:43.880062 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:43.879842    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 31 14:49:43.880164 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:43.879782    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 31 14:49:43.882368 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:43.882348    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb9599594-9n865"
Jul 31 14:49:43.885006 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:43.884983    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 31 14:49:43.885369 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:43.885350    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 31 14:49:43.885614 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:43.885594    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 31 14:49:43.885698 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:43.885647    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 31 14:49:43.886722 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:43.886703    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59d4fdc6b8-r7hdk"]
Jul 31 14:49:43.897018 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:43.896998    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb9599594-9n865"]
Jul 31 14:49:43.999892 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:43.999855    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2d0e735b-3b55-4687-a738-06c43f4eaefc-hub\") pod \"cluster-proxy-proxy-agent-6fb9599594-9n865\" (UID: \"2d0e735b-3b55-4687-a738-06c43f4eaefc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb9599594-9n865"
Jul 31 14:49:43.999892 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:43.999895    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm87w\" (UniqueName: \"kubernetes.io/projected/2d0e735b-3b55-4687-a738-06c43f4eaefc-kube-api-access-nm87w\") pod \"cluster-proxy-proxy-agent-6fb9599594-9n865\" (UID: \"2d0e735b-3b55-4687-a738-06c43f4eaefc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb9599594-9n865"
Jul 31 14:49:44.000122 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:43.999920    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx4l4\" (UniqueName: \"kubernetes.io/projected/6ca9a551-691d-4add-8a12-52365521967f-kube-api-access-tx4l4\") pod \"managed-serviceaccount-addon-agent-59d4fdc6b8-r7hdk\" (UID: \"6ca9a551-691d-4add-8a12-52365521967f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59d4fdc6b8-r7hdk"
Jul 31 14:49:44.000122 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:43.999969    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d0e735b-3b55-4687-a738-06c43f4eaefc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6fb9599594-9n865\" (UID: \"2d0e735b-3b55-4687-a738-06c43f4eaefc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb9599594-9n865"
Jul 31 14:49:44.000122 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:44.000009    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6ca9a551-691d-4add-8a12-52365521967f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-59d4fdc6b8-r7hdk\" (UID: \"6ca9a551-691d-4add-8a12-52365521967f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59d4fdc6b8-r7hdk"
Jul 31 14:49:44.000122 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:44.000097    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2d0e735b-3b55-4687-a738-06c43f4eaefc-ca\") pod \"cluster-proxy-proxy-agent-6fb9599594-9n865\" (UID: \"2d0e735b-3b55-4687-a738-06c43f4eaefc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb9599594-9n865"
Jul 31 14:49:44.000299 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:44.000164    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0e735b-3b55-4687-a738-06c43f4eaefc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6fb9599594-9n865\" (UID: \"2d0e735b-3b55-4687-a738-06c43f4eaefc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb9599594-9n865"
Jul 31 14:49:44.000299 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:44.000210    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2d0e735b-3b55-4687-a738-06c43f4eaefc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6fb9599594-9n865\" (UID: \"2d0e735b-3b55-4687-a738-06c43f4eaefc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb9599594-9n865"
Jul 31 14:49:44.101317 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:44.101282    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2d0e735b-3b55-4687-a738-06c43f4eaefc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6fb9599594-9n865\" (UID: \"2d0e735b-3b55-4687-a738-06c43f4eaefc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb9599594-9n865"
Jul 31 14:49:44.101317 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:44.101315    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2d0e735b-3b55-4687-a738-06c43f4eaefc-hub\") pod \"cluster-proxy-proxy-agent-6fb9599594-9n865\" (UID: \"2d0e735b-3b55-4687-a738-06c43f4eaefc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb9599594-9n865"
Jul 31 14:49:44.101570 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:44.101337    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm87w\" (UniqueName: \"kubernetes.io/projected/2d0e735b-3b55-4687-a738-06c43f4eaefc-kube-api-access-nm87w\") pod \"cluster-proxy-proxy-agent-6fb9599594-9n865\" (UID: \"2d0e735b-3b55-4687-a738-06c43f4eaefc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb9599594-9n865"
Jul 31 14:49:44.101570 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:44.101356    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx4l4\" (UniqueName: \"kubernetes.io/projected/6ca9a551-691d-4add-8a12-52365521967f-kube-api-access-tx4l4\") pod \"managed-serviceaccount-addon-agent-59d4fdc6b8-r7hdk\" (UID: \"6ca9a551-691d-4add-8a12-52365521967f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59d4fdc6b8-r7hdk"
Jul 31 14:49:44.101570 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:44.101408    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d0e735b-3b55-4687-a738-06c43f4eaefc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6fb9599594-9n865\" (UID: \"2d0e735b-3b55-4687-a738-06c43f4eaefc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb9599594-9n865"
Jul 31 14:49:44.101570 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:44.101453    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6ca9a551-691d-4add-8a12-52365521967f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-59d4fdc6b8-r7hdk\" (UID: \"6ca9a551-691d-4add-8a12-52365521967f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59d4fdc6b8-r7hdk"
Jul 31 14:49:44.101824 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:44.101611    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2d0e735b-3b55-4687-a738-06c43f4eaefc-ca\") pod \"cluster-proxy-proxy-agent-6fb9599594-9n865\" (UID: \"2d0e735b-3b55-4687-a738-06c43f4eaefc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb9599594-9n865"
Jul 31 14:49:44.101824 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:44.101671    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0e735b-3b55-4687-a738-06c43f4eaefc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6fb9599594-9n865\" (UID: \"2d0e735b-3b55-4687-a738-06c43f4eaefc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb9599594-9n865"
Jul 31 14:49:44.102447 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:44.102418    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0e735b-3b55-4687-a738-06c43f4eaefc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6fb9599594-9n865\" (UID: \"2d0e735b-3b55-4687-a738-06c43f4eaefc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb9599594-9n865"
Jul 31 14:49:44.104142 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:44.104113    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d0e735b-3b55-4687-a738-06c43f4eaefc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6fb9599594-9n865\" (UID: \"2d0e735b-3b55-4687-a738-06c43f4eaefc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb9599594-9n865"
Jul 31 14:49:44.104490 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:44.104468    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2d0e735b-3b55-4687-a738-06c43f4eaefc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6fb9599594-9n865\" (UID: \"2d0e735b-3b55-4687-a738-06c43f4eaefc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb9599594-9n865"
Jul 31 14:49:44.104576 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:44.104546    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2d0e735b-3b55-4687-a738-06c43f4eaefc-hub\") pod \"cluster-proxy-proxy-agent-6fb9599594-9n865\" (UID: \"2d0e735b-3b55-4687-a738-06c43f4eaefc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb9599594-9n865"
Jul 31 14:49:44.104786 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:44.104761    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2d0e735b-3b55-4687-a738-06c43f4eaefc-ca\") pod \"cluster-proxy-proxy-agent-6fb9599594-9n865\" (UID: \"2d0e735b-3b55-4687-a738-06c43f4eaefc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb9599594-9n865"
Jul 31 14:49:44.104918 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:44.104904    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6ca9a551-691d-4add-8a12-52365521967f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-59d4fdc6b8-r7hdk\" (UID: \"6ca9a551-691d-4add-8a12-52365521967f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59d4fdc6b8-r7hdk"
Jul 31 14:49:44.116151 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:44.116123    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm87w\" (UniqueName: \"kubernetes.io/projected/2d0e735b-3b55-4687-a738-06c43f4eaefc-kube-api-access-nm87w\") pod \"cluster-proxy-proxy-agent-6fb9599594-9n865\" (UID: \"2d0e735b-3b55-4687-a738-06c43f4eaefc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb9599594-9n865"
Jul 31 14:49:44.118243 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:44.118223    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx4l4\" (UniqueName: \"kubernetes.io/projected/6ca9a551-691d-4add-8a12-52365521967f-kube-api-access-tx4l4\") pod \"managed-serviceaccount-addon-agent-59d4fdc6b8-r7hdk\" (UID: \"6ca9a551-691d-4add-8a12-52365521967f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59d4fdc6b8-r7hdk"
Jul 31 14:49:44.189747 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:44.189632    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59d4fdc6b8-r7hdk"
Jul 31 14:49:44.196645 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:44.196621    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb9599594-9n865"
Jul 31 14:49:44.346075 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:44.346044    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59d4fdc6b8-r7hdk"]
Jul 31 14:49:44.349119 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:49:44.349094    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ca9a551_691d_4add_8a12_52365521967f.slice/crio-fd33ffbcda812ca292555d695cf21eaf236752c48b8796582530a3545e33fa81 WatchSource:0}: Error finding container fd33ffbcda812ca292555d695cf21eaf236752c48b8796582530a3545e33fa81: Status 404 returned error can't find the container with id fd33ffbcda812ca292555d695cf21eaf236752c48b8796582530a3545e33fa81
Jul 31 14:49:44.369399 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:44.369362    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb9599594-9n865"]
Jul 31 14:49:44.372664 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:49:44.372636    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d0e735b_3b55_4687_a738_06c43f4eaefc.slice/crio-b0a75bf97d5944926c3b1aefb7e0646dd369ceab3a7a757a4322ab89ffc1305a WatchSource:0}: Error finding container b0a75bf97d5944926c3b1aefb7e0646dd369ceab3a7a757a4322ab89ffc1305a: Status 404 returned error can't find the container with id b0a75bf97d5944926c3b1aefb7e0646dd369ceab3a7a757a4322ab89ffc1305a
Jul 31 14:49:44.546533 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:44.546439    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59d4fdc6b8-r7hdk" event={"ID":"6ca9a551-691d-4add-8a12-52365521967f","Type":"ContainerStarted","Data":"fd33ffbcda812ca292555d695cf21eaf236752c48b8796582530a3545e33fa81"}
Jul 31 14:49:44.547438 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:44.547416    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb9599594-9n865" event={"ID":"2d0e735b-3b55-4687-a738-06c43f4eaefc","Type":"ContainerStarted","Data":"b0a75bf97d5944926c3b1aefb7e0646dd369ceab3a7a757a4322ab89ffc1305a"}
Jul 31 14:49:45.714753 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:45.714589    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e1cfecbe-4155-4b24-aad5-2277fd0c350e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-m4hkx\" (UID: \"e1cfecbe-4155-4b24-aad5-2277fd0c350e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m4hkx"
Jul 31 14:49:45.728679 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:45.728617    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e1cfecbe-4155-4b24-aad5-2277fd0c350e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-m4hkx\" (UID: \"e1cfecbe-4155-4b24-aad5-2277fd0c350e\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m4hkx"
Jul 31 14:49:45.900500 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:45.900322    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gc59x\""
Jul 31 14:49:45.905971 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:45.905942    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m4hkx"
Jul 31 14:49:46.120509 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:46.120376    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m4hkx"]
Jul 31 14:49:46.129686 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:49:46.129660    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1cfecbe_4155_4b24_aad5_2277fd0c350e.slice/crio-698808729eb79f37f2742479074fee16f444929c1dbc1502aed5a1d6862c6936 WatchSource:0}: Error finding container 698808729eb79f37f2742479074fee16f444929c1dbc1502aed5a1d6862c6936: Status 404 returned error can't find the container with id 698808729eb79f37f2742479074fee16f444929c1dbc1502aed5a1d6862c6936
Jul 31 14:49:46.511618 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:46.511583    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2mr24"
Jul 31 14:49:46.558117 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:46.558077    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m4hkx" event={"ID":"e1cfecbe-4155-4b24-aad5-2277fd0c350e","Type":"ContainerStarted","Data":"698808729eb79f37f2742479074fee16f444929c1dbc1502aed5a1d6862c6936"}
Jul 31 14:49:50.579372 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:50.579331    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb9599594-9n865" event={"ID":"2d0e735b-3b55-4687-a738-06c43f4eaefc","Type":"ContainerStarted","Data":"544a5c363852623ac8ac8e737cd53bf6003c13aa44f02fb1555be5413c42f543"}
Jul 31 14:49:50.579872 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:50.579382    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb9599594-9n865" event={"ID":"2d0e735b-3b55-4687-a738-06c43f4eaefc","Type":"ContainerStarted","Data":"1949915d400be3bbcca4d32f600245a14323f79f6c63c21047d2a9340521c846"}
Jul 31 14:49:50.579872 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:50.579397    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb9599594-9n865" event={"ID":"2d0e735b-3b55-4687-a738-06c43f4eaefc","Type":"ContainerStarted","Data":"fc083d36dd8b4b07bde907c4191dd3a8cfcc343fa67c832b69dadabff2f18b01"}
Jul 31 14:49:50.580938 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:50.580911    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59d4fdc6b8-r7hdk" event={"ID":"6ca9a551-691d-4add-8a12-52365521967f","Type":"ContainerStarted","Data":"e1b3ac32087da5d86529b313ddd441902929681396951bc42152ddd1881a9d0e"}
Jul 31 14:49:50.582287 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:50.582267    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m4hkx" event={"ID":"e1cfecbe-4155-4b24-aad5-2277fd0c350e","Type":"ContainerStarted","Data":"8c1ac7b688c4d2be1115116e336219aa7a740c372ccfd36afe5fe928b34365b3"}
Jul 31 14:49:50.604114 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:50.604067    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fb9599594-9n865" podStartSLOduration=2.124938507 podStartE2EDuration="7.604052501s" totalImagesPullingTime="5.479113994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:50.603334351 +0000 UTC m=+86.914891972" watchObservedRunningTime="2026-07-31 14:49:50.604052501 +0000 UTC m=+86.915610123"
Jul 31 14:49:50.623145 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:50.623081    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-m4hkx" podStartSLOduration=33.896383885 podStartE2EDuration="37.623065868s" totalImagesPullingTime="3.726681983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:50.621692488 +0000 UTC m=+86.933250109" watchObservedRunningTime="2026-07-31 14:49:50.623065868 +0000 UTC m=+86.934623489"
Jul 31 14:49:50.640862 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:50.640813    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59d4fdc6b8-r7hdk" podStartSLOduration=2.192323457 podStartE2EDuration="7.640795462s" totalImagesPullingTime="5.448472005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:50.640106313 +0000 UTC m=+86.951663934" watchObservedRunningTime="2026-07-31 14:49:50.640795462 +0000 UTC m=+86.952353084"
Jul 31 14:49:59.952914 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:59.952884    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qss7j"]
Jul 31 14:49:59.955778 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:59.955759    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:49:59.960923 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:59.960901    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lcpl5\""
Jul 31 14:49:59.961052 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:59.960900    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 31 14:49:59.961336 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:59.961318    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 31 14:49:59.961899 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:59.961882    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 31 14:49:59.962382 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:49:59.962112    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 31 14:50:00.127916 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.127879    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/59378969-47f3-45bd-b6f5-0407ae63516c-root\") pod \"node-exporter-qss7j\" (UID: \"59378969-47f3-45bd-b6f5-0407ae63516c\") " pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.128104 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.127935    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59378969-47f3-45bd-b6f5-0407ae63516c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qss7j\" (UID: \"59378969-47f3-45bd-b6f5-0407ae63516c\") " pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.128104 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.127961    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59378969-47f3-45bd-b6f5-0407ae63516c-node-exporter-tls\") pod \"node-exporter-qss7j\" (UID: \"59378969-47f3-45bd-b6f5-0407ae63516c\") " pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.128104 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.128032    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59378969-47f3-45bd-b6f5-0407ae63516c-metrics-client-ca\") pod \"node-exporter-qss7j\" (UID: \"59378969-47f3-45bd-b6f5-0407ae63516c\") " pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.128104 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.128061    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/59378969-47f3-45bd-b6f5-0407ae63516c-node-exporter-wtmp\") pod \"node-exporter-qss7j\" (UID: \"59378969-47f3-45bd-b6f5-0407ae63516c\") " pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.128104 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.128084    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59378969-47f3-45bd-b6f5-0407ae63516c-sys\") pod \"node-exporter-qss7j\" (UID: \"59378969-47f3-45bd-b6f5-0407ae63516c\") " pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.128104 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.128105    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/59378969-47f3-45bd-b6f5-0407ae63516c-node-exporter-accelerators-collector-config\") pod \"node-exporter-qss7j\" (UID: \"59378969-47f3-45bd-b6f5-0407ae63516c\") " pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.128359 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.128145    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/59378969-47f3-45bd-b6f5-0407ae63516c-node-exporter-textfile\") pod \"node-exporter-qss7j\" (UID: \"59378969-47f3-45bd-b6f5-0407ae63516c\") " pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.128359 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.128178    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55p9b\" (UniqueName: \"kubernetes.io/projected/59378969-47f3-45bd-b6f5-0407ae63516c-kube-api-access-55p9b\") pod \"node-exporter-qss7j\" (UID: \"59378969-47f3-45bd-b6f5-0407ae63516c\") " pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.228585 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.228499    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/59378969-47f3-45bd-b6f5-0407ae63516c-node-exporter-textfile\") pod \"node-exporter-qss7j\" (UID: \"59378969-47f3-45bd-b6f5-0407ae63516c\") " pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.228585 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.228542    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55p9b\" (UniqueName: \"kubernetes.io/projected/59378969-47f3-45bd-b6f5-0407ae63516c-kube-api-access-55p9b\") pod \"node-exporter-qss7j\" (UID: \"59378969-47f3-45bd-b6f5-0407ae63516c\") " pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.228585 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.228560    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/59378969-47f3-45bd-b6f5-0407ae63516c-root\") pod \"node-exporter-qss7j\" (UID: \"59378969-47f3-45bd-b6f5-0407ae63516c\") " pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.228904 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.228604    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59378969-47f3-45bd-b6f5-0407ae63516c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qss7j\" (UID: \"59378969-47f3-45bd-b6f5-0407ae63516c\") " pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.228904 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.228631    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59378969-47f3-45bd-b6f5-0407ae63516c-node-exporter-tls\") pod \"node-exporter-qss7j\" (UID: \"59378969-47f3-45bd-b6f5-0407ae63516c\") " pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.228904 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.228664    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59378969-47f3-45bd-b6f5-0407ae63516c-metrics-client-ca\") pod \"node-exporter-qss7j\" (UID: \"59378969-47f3-45bd-b6f5-0407ae63516c\") " pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.228904 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.228694    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/59378969-47f3-45bd-b6f5-0407ae63516c-node-exporter-wtmp\") pod \"node-exporter-qss7j\" (UID: \"59378969-47f3-45bd-b6f5-0407ae63516c\") " pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.228904 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.228767    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/59378969-47f3-45bd-b6f5-0407ae63516c-root\") pod \"node-exporter-qss7j\" (UID: \"59378969-47f3-45bd-b6f5-0407ae63516c\") " pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.228904 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:50:00.228785    2559 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 31 14:50:00.228904 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.228844    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/59378969-47f3-45bd-b6f5-0407ae63516c-node-exporter-wtmp\") pod \"node-exporter-qss7j\" (UID: \"59378969-47f3-45bd-b6f5-0407ae63516c\") " pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.228904 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.228850    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59378969-47f3-45bd-b6f5-0407ae63516c-sys\") pod \"node-exporter-qss7j\" (UID: \"59378969-47f3-45bd-b6f5-0407ae63516c\") " pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.228904 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:50:00.228863    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59378969-47f3-45bd-b6f5-0407ae63516c-node-exporter-tls podName:59378969-47f3-45bd-b6f5-0407ae63516c nodeName:}" failed. No retries permitted until 2026-07-31 14:50:00.728842881 +0000 UTC m=+97.040400481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/59378969-47f3-45bd-b6f5-0407ae63516c-node-exporter-tls") pod "node-exporter-qss7j" (UID: "59378969-47f3-45bd-b6f5-0407ae63516c") : secret "node-exporter-tls" not found
Jul 31 14:50:00.229350 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.228909    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/59378969-47f3-45bd-b6f5-0407ae63516c-node-exporter-accelerators-collector-config\") pod \"node-exporter-qss7j\" (UID: \"59378969-47f3-45bd-b6f5-0407ae63516c\") " pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.229350 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.228916    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59378969-47f3-45bd-b6f5-0407ae63516c-sys\") pod \"node-exporter-qss7j\" (UID: \"59378969-47f3-45bd-b6f5-0407ae63516c\") " pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.229350 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.228987    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/59378969-47f3-45bd-b6f5-0407ae63516c-node-exporter-textfile\") pod \"node-exporter-qss7j\" (UID: \"59378969-47f3-45bd-b6f5-0407ae63516c\") " pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.229350 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.229331    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59378969-47f3-45bd-b6f5-0407ae63516c-metrics-client-ca\") pod \"node-exporter-qss7j\" (UID: \"59378969-47f3-45bd-b6f5-0407ae63516c\") " pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.229478 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.229400    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/59378969-47f3-45bd-b6f5-0407ae63516c-node-exporter-accelerators-collector-config\") pod \"node-exporter-qss7j\" (UID: \"59378969-47f3-45bd-b6f5-0407ae63516c\") " pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.231274 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.231252    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59378969-47f3-45bd-b6f5-0407ae63516c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qss7j\" (UID: \"59378969-47f3-45bd-b6f5-0407ae63516c\") " pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.243548 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.243525    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55p9b\" (UniqueName: \"kubernetes.io/projected/59378969-47f3-45bd-b6f5-0407ae63516c-kube-api-access-55p9b\") pod \"node-exporter-qss7j\" (UID: \"59378969-47f3-45bd-b6f5-0407ae63516c\") " pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.733586 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.733547    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59378969-47f3-45bd-b6f5-0407ae63516c-node-exporter-tls\") pod \"node-exporter-qss7j\" (UID: \"59378969-47f3-45bd-b6f5-0407ae63516c\") " pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.735957 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.735928    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59378969-47f3-45bd-b6f5-0407ae63516c-node-exporter-tls\") pod \"node-exporter-qss7j\" (UID: \"59378969-47f3-45bd-b6f5-0407ae63516c\") " pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.865456 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:00.865420    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qss7j"
Jul 31 14:50:00.877058 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:50:00.877021    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59378969_47f3_45bd_b6f5_0407ae63516c.slice/crio-8aa622e6bed1dd3f6557976743064db232e5d46a48527b406b0b107298c2e94a WatchSource:0}: Error finding container 8aa622e6bed1dd3f6557976743064db232e5d46a48527b406b0b107298c2e94a: Status 404 returned error can't find the container with id 8aa622e6bed1dd3f6557976743064db232e5d46a48527b406b0b107298c2e94a
Jul 31 14:50:01.623611 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:01.623572    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qss7j" event={"ID":"59378969-47f3-45bd-b6f5-0407ae63516c","Type":"ContainerStarted","Data":"8aa622e6bed1dd3f6557976743064db232e5d46a48527b406b0b107298c2e94a"}
Jul 31 14:50:02.628480 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:02.628442    2559 generic.go:356] "Generic (PLEG): container finished" podID="59378969-47f3-45bd-b6f5-0407ae63516c" containerID="ed77554d76528a142c500dddbc3bc407d27e946d5d6c69ca9bd3ad1a6e6b79b0" exitCode=0
Jul 31 14:50:02.628876 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:02.628489    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qss7j" event={"ID":"59378969-47f3-45bd-b6f5-0407ae63516c","Type":"ContainerDied","Data":"ed77554d76528a142c500dddbc3bc407d27e946d5d6c69ca9bd3ad1a6e6b79b0"}
Jul 31 14:50:03.634326 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:03.634288    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qss7j" event={"ID":"59378969-47f3-45bd-b6f5-0407ae63516c","Type":"ContainerStarted","Data":"fd5d03e05b11b63fad064a65a2731534b997b050bfc566112d3326e29c32ee47"}
Jul 31 14:50:03.634710 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:03.634342    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qss7j" event={"ID":"59378969-47f3-45bd-b6f5-0407ae63516c","Type":"ContainerStarted","Data":"bd2dfc71b128bac6bc790ac22135f3020cf1934a88014daa8a993c0686a0a9cc"}
Jul 31 14:50:03.658135 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:03.658085    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qss7j" podStartSLOduration=3.614512823 podStartE2EDuration="4.658064634s" totalImagesPullingTime="970.903652ms" totalInitContainerRuntime="72.648159ms" isStatefulPod=true podCreationTimestamp="2026-07-31 14:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:50:03.656807445 +0000 UTC m=+99.968365067" watchObservedRunningTime="2026-07-31 14:50:03.658064634 +0000 UTC m=+99.969622257"
Jul 31 14:50:07.511399 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:07.511367    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6vfsr"
Jul 31 14:50:28.718672 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:28.718635    2559 generic.go:356] "Generic (PLEG): container finished" podID="1380fa66-45b6-48b5-aee6-18f70ca49ebf" containerID="5936c893de825417ab34b9f5d62dfbc46b3a56ee200df026f81b0fd49738b08f" exitCode=0
Jul 31 14:50:28.719133 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:28.718709    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-c6dsw" event={"ID":"1380fa66-45b6-48b5-aee6-18f70ca49ebf","Type":"ContainerDied","Data":"5936c893de825417ab34b9f5d62dfbc46b3a56ee200df026f81b0fd49738b08f"}
Jul 31 14:50:28.719133 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:28.719020    2559 scope.go:117] "RemoveContainer" containerID="5936c893de825417ab34b9f5d62dfbc46b3a56ee200df026f81b0fd49738b08f"
Jul 31 14:50:29.723587 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:29.723553    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-c6dsw" event={"ID":"1380fa66-45b6-48b5-aee6-18f70ca49ebf","Type":"ContainerStarted","Data":"b191d018197e8fcd08942712f67dd780410b18164ddc37ec139d9658c9881aad"}
Jul 31 14:50:33.739679 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:33.739642    2559 generic.go:356] "Generic (PLEG): container finished" podID="473e45a2-a84a-4a9f-8950-959e755baefa" containerID="44b805b8a24605b9f3b4714a8246313331e4f52d0b67f7f3071657ea20c31bfe" exitCode=0
Jul 31 14:50:33.739679 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:33.739679    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mngjp" event={"ID":"473e45a2-a84a-4a9f-8950-959e755baefa","Type":"ContainerDied","Data":"44b805b8a24605b9f3b4714a8246313331e4f52d0b67f7f3071657ea20c31bfe"}
Jul 31 14:50:33.740198 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:33.739985    2559 scope.go:117] "RemoveContainer" containerID="44b805b8a24605b9f3b4714a8246313331e4f52d0b67f7f3071657ea20c31bfe"
Jul 31 14:50:33.894170 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:33.894128    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355d9cbb-1867-49cc-8ec2-564b4aa6daea-metrics-certs\") pod \"network-metrics-daemon-b45r4\" (UID: \"355d9cbb-1867-49cc-8ec2-564b4aa6daea\") " pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:50:33.896597 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:33.896573    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355d9cbb-1867-49cc-8ec2-564b4aa6daea-metrics-certs\") pod \"network-metrics-daemon-b45r4\" (UID: \"355d9cbb-1867-49cc-8ec2-564b4aa6daea\") " pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:50:33.938303 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:33.938269    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jrzg7\""
Jul 31 14:50:33.944787 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:33.944768    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b45r4"
Jul 31 14:50:34.076411 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:34.076378    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b45r4"]
Jul 31 14:50:34.079524 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:50:34.079490    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355d9cbb_1867_49cc_8ec2_564b4aa6daea.slice/crio-041667ec37798ba43101b4b4c6bc09bee94fffc8b59a591ca521602f6f76768d WatchSource:0}: Error finding container 041667ec37798ba43101b4b4c6bc09bee94fffc8b59a591ca521602f6f76768d: Status 404 returned error can't find the container with id 041667ec37798ba43101b4b4c6bc09bee94fffc8b59a591ca521602f6f76768d
Jul 31 14:50:34.745240 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:34.745199    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-mngjp" event={"ID":"473e45a2-a84a-4a9f-8950-959e755baefa","Type":"ContainerStarted","Data":"1a3560f5dca96c3b0b8e6a4bbd5699c30b94627609a357cc666da2b634af3980"}
Jul 31 14:50:34.746404 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:34.746378    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b45r4" event={"ID":"355d9cbb-1867-49cc-8ec2-564b4aa6daea","Type":"ContainerStarted","Data":"041667ec37798ba43101b4b4c6bc09bee94fffc8b59a591ca521602f6f76768d"}
Jul 31 14:50:35.751121 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:35.751087    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b45r4" event={"ID":"355d9cbb-1867-49cc-8ec2-564b4aa6daea","Type":"ContainerStarted","Data":"2224bafa0578a1249a6e9f8116e3aebedf9f14a9f2aba46a64cb20237db25ece"}
Jul 31 14:50:35.751500 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:35.751138    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b45r4" event={"ID":"355d9cbb-1867-49cc-8ec2-564b4aa6daea","Type":"ContainerStarted","Data":"6fb8747133370607ea686938398656cf699494d61813d9e5e3ec21a6c1353e47"}
Jul 31 14:50:35.770109 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:50:35.770061    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-b45r4" podStartSLOduration=130.924261044 podStartE2EDuration="2m11.770042536s" totalImagesPullingTime="845.781492ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:50:35.769264325 +0000 UTC m=+132.080821967" watchObservedRunningTime="2026-07-31 14:50:35.770042536 +0000 UTC m=+132.081600157"
Jul 31 14:53:24.080306 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:53:24.080275    2559 kubelet.go:1648] "Image garbage collection succeeded"
Jul 31 14:55:28.222347 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.222311    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-fcc9dd4db-48crh"]
Jul 31 14:55:28.224276 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.224260    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-fcc9dd4db-48crh"
Jul 31 14:55:28.227092 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.227062    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 31 14:55:28.228056 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.228037    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 31 14:55:28.228146 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.228084    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-49kr9\""
Jul 31 14:55:28.228361 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.228345    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 31 14:55:28.240419 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.240386    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-fcc9dd4db-48crh"]
Jul 31 14:55:28.260366 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.260336    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-wx8q8"]
Jul 31 14:55:28.262713 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.262690    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-wx8q8"
Jul 31 14:55:28.265902 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.265881    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 31 14:55:28.265902 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.265896    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-z6qcv\""
Jul 31 14:55:28.275272 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.275248    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-wx8q8"]
Jul 31 14:55:28.282554 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.282528    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1f4263e6-2f78-4867-8087-5be7e0ba9ea3-data\") pod \"seaweedfs-64568bcd49-wx8q8\" (UID: \"1f4263e6-2f78-4867-8087-5be7e0ba9ea3\") " pod="kserve/seaweedfs-64568bcd49-wx8q8"
Jul 31 14:55:28.282647 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.282585    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4tc5\" (UniqueName: \"kubernetes.io/projected/1f4263e6-2f78-4867-8087-5be7e0ba9ea3-kube-api-access-k4tc5\") pod \"seaweedfs-64568bcd49-wx8q8\" (UID: \"1f4263e6-2f78-4867-8087-5be7e0ba9ea3\") " pod="kserve/seaweedfs-64568bcd49-wx8q8"
Jul 31 14:55:28.282685 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.282653    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1bba895-2fc9-41d1-a948-6f94043d3e90-cert\") pod \"kserve-controller-manager-fcc9dd4db-48crh\" (UID: \"e1bba895-2fc9-41d1-a948-6f94043d3e90\") " pod="kserve/kserve-controller-manager-fcc9dd4db-48crh"
Jul 31 14:55:28.282722 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.282695    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxv6j\" (UniqueName: \"kubernetes.io/projected/e1bba895-2fc9-41d1-a948-6f94043d3e90-kube-api-access-wxv6j\") pod \"kserve-controller-manager-fcc9dd4db-48crh\" (UID: \"e1bba895-2fc9-41d1-a948-6f94043d3e90\") " pod="kserve/kserve-controller-manager-fcc9dd4db-48crh"
Jul 31 14:55:28.383596 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.383550    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4tc5\" (UniqueName: \"kubernetes.io/projected/1f4263e6-2f78-4867-8087-5be7e0ba9ea3-kube-api-access-k4tc5\") pod \"seaweedfs-64568bcd49-wx8q8\" (UID: \"1f4263e6-2f78-4867-8087-5be7e0ba9ea3\") " pod="kserve/seaweedfs-64568bcd49-wx8q8"
Jul 31 14:55:28.383596 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.383605    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1bba895-2fc9-41d1-a948-6f94043d3e90-cert\") pod \"kserve-controller-manager-fcc9dd4db-48crh\" (UID: \"e1bba895-2fc9-41d1-a948-6f94043d3e90\") " pod="kserve/kserve-controller-manager-fcc9dd4db-48crh"
Jul 31 14:55:28.383882 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.383633    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxv6j\" (UniqueName: \"kubernetes.io/projected/e1bba895-2fc9-41d1-a948-6f94043d3e90-kube-api-access-wxv6j\") pod \"kserve-controller-manager-fcc9dd4db-48crh\" (UID: \"e1bba895-2fc9-41d1-a948-6f94043d3e90\") " pod="kserve/kserve-controller-manager-fcc9dd4db-48crh"
Jul 31 14:55:28.383882 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.383665    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1f4263e6-2f78-4867-8087-5be7e0ba9ea3-data\") pod \"seaweedfs-64568bcd49-wx8q8\" (UID: \"1f4263e6-2f78-4867-8087-5be7e0ba9ea3\") " pod="kserve/seaweedfs-64568bcd49-wx8q8"
Jul 31 14:55:28.384047 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.384029    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1f4263e6-2f78-4867-8087-5be7e0ba9ea3-data\") pod \"seaweedfs-64568bcd49-wx8q8\" (UID: \"1f4263e6-2f78-4867-8087-5be7e0ba9ea3\") " pod="kserve/seaweedfs-64568bcd49-wx8q8"
Jul 31 14:55:28.386431 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.386397    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1bba895-2fc9-41d1-a948-6f94043d3e90-cert\") pod \"kserve-controller-manager-fcc9dd4db-48crh\" (UID: \"e1bba895-2fc9-41d1-a948-6f94043d3e90\") " pod="kserve/kserve-controller-manager-fcc9dd4db-48crh"
Jul 31 14:55:28.397458 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.397419    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4tc5\" (UniqueName: \"kubernetes.io/projected/1f4263e6-2f78-4867-8087-5be7e0ba9ea3-kube-api-access-k4tc5\") pod \"seaweedfs-64568bcd49-wx8q8\" (UID: \"1f4263e6-2f78-4867-8087-5be7e0ba9ea3\") " pod="kserve/seaweedfs-64568bcd49-wx8q8"
Jul 31 14:55:28.402956 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.402928    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxv6j\" (UniqueName: \"kubernetes.io/projected/e1bba895-2fc9-41d1-a948-6f94043d3e90-kube-api-access-wxv6j\") pod \"kserve-controller-manager-fcc9dd4db-48crh\" (UID: \"e1bba895-2fc9-41d1-a948-6f94043d3e90\") " pod="kserve/kserve-controller-manager-fcc9dd4db-48crh"
Jul 31 14:55:28.534865 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.534772    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-fcc9dd4db-48crh"
Jul 31 14:55:28.571183 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.571148    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-wx8q8"
Jul 31 14:55:28.673893 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.673861    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-fcc9dd4db-48crh"]
Jul 31 14:55:28.677281 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:55:28.677247    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1bba895_2fc9_41d1_a948_6f94043d3e90.slice/crio-f542e72e36df992fbf6ce86cf3ab12fa4fb152655d1b3b8f1c11884903e6b040 WatchSource:0}: Error finding container f542e72e36df992fbf6ce86cf3ab12fa4fb152655d1b3b8f1c11884903e6b040: Status 404 returned error can't find the container with id f542e72e36df992fbf6ce86cf3ab12fa4fb152655d1b3b8f1c11884903e6b040
Jul 31 14:55:28.678957 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.678938    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 14:55:28.720327 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.720295    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-wx8q8"]
Jul 31 14:55:28.725544 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:55:28.725520    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f4263e6_2f78_4867_8087_5be7e0ba9ea3.slice/crio-9fbaa77f5df9b4d320cdab2a8377bf6104974dd49f431a527fb479bed4ca7993 WatchSource:0}: Error finding container 9fbaa77f5df9b4d320cdab2a8377bf6104974dd49f431a527fb479bed4ca7993: Status 404 returned error can't find the container with id 9fbaa77f5df9b4d320cdab2a8377bf6104974dd49f431a527fb479bed4ca7993
Jul 31 14:55:28.754484 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.754452    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-wx8q8" event={"ID":"1f4263e6-2f78-4867-8087-5be7e0ba9ea3","Type":"ContainerStarted","Data":"9fbaa77f5df9b4d320cdab2a8377bf6104974dd49f431a527fb479bed4ca7993"}
Jul 31 14:55:28.755484 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:28.755460    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-fcc9dd4db-48crh" event={"ID":"e1bba895-2fc9-41d1-a948-6f94043d3e90","Type":"ContainerStarted","Data":"f542e72e36df992fbf6ce86cf3ab12fa4fb152655d1b3b8f1c11884903e6b040"}
Jul 31 14:55:32.774608 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:32.774579    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-fcc9dd4db-48crh" event={"ID":"e1bba895-2fc9-41d1-a948-6f94043d3e90","Type":"ContainerStarted","Data":"27b1530a69677282218502180278dfa1fa9662377ff98d7205ed2918c723bac8"}
Jul 31 14:55:32.774999 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:32.774653    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-fcc9dd4db-48crh"
Jul 31 14:55:32.793394 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:32.793342    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-fcc9dd4db-48crh" podStartSLOduration=0.802376203 podStartE2EDuration="4.793321449s" totalImagesPullingTime="3.990945246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:55:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:55:32.792972291 +0000 UTC m=+429.104529913" watchObservedRunningTime="2026-07-31 14:55:32.793321449 +0000 UTC m=+429.104879070"
Jul 31 14:55:33.779026 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:33.778987    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-wx8q8" event={"ID":"1f4263e6-2f78-4867-8087-5be7e0ba9ea3","Type":"ContainerStarted","Data":"9869deebb011986a5dcc69e9b8d7b57fa7856f34ce4c9786151e6bf3c747a665"}
Jul 31 14:55:33.779492 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:33.779123    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-wx8q8"
Jul 31 14:55:33.797632 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:33.797569    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-wx8q8" podStartSLOduration=1.767745922 podStartE2EDuration="5.797556646s" totalImagesPullingTime="4.029810724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:55:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:55:33.796563526 +0000 UTC m=+430.108121147" watchObservedRunningTime="2026-07-31 14:55:33.797556646 +0000 UTC m=+430.109114267"
Jul 31 14:55:39.785145 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:55:39.785106    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-wx8q8"
Jul 31 14:56:03.784158 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:03.784121    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-fcc9dd4db-48crh"
Jul 31 14:56:12.940747 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:12.940704    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-fcc9dd4db-48crh"]
Jul 31 14:56:12.941300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:12.940974    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-fcc9dd4db-48crh" podUID="e1bba895-2fc9-41d1-a948-6f94043d3e90" containerName="manager" containerID="cri-o://27b1530a69677282218502180278dfa1fa9662377ff98d7205ed2918c723bac8" gracePeriod=10
Jul 31 14:56:12.992448 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:12.992410    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-fcc9dd4db-4t79k"]
Jul 31 14:56:12.994604 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:12.994579    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-fcc9dd4db-4t79k"
Jul 31 14:56:13.011848 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.011825    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-fcc9dd4db-4t79k"]
Jul 31 14:56:13.118309 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.118273    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh7cf\" (UniqueName: \"kubernetes.io/projected/2cde6c4f-b5ff-4e16-a860-77959ec50852-kube-api-access-hh7cf\") pod \"kserve-controller-manager-fcc9dd4db-4t79k\" (UID: \"2cde6c4f-b5ff-4e16-a860-77959ec50852\") " pod="kserve/kserve-controller-manager-fcc9dd4db-4t79k"
Jul 31 14:56:13.118447 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.118395    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2cde6c4f-b5ff-4e16-a860-77959ec50852-cert\") pod \"kserve-controller-manager-fcc9dd4db-4t79k\" (UID: \"2cde6c4f-b5ff-4e16-a860-77959ec50852\") " pod="kserve/kserve-controller-manager-fcc9dd4db-4t79k"
Jul 31 14:56:13.178675 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.178648    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-fcc9dd4db-48crh"
Jul 31 14:56:13.219794 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.219663    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh7cf\" (UniqueName: \"kubernetes.io/projected/2cde6c4f-b5ff-4e16-a860-77959ec50852-kube-api-access-hh7cf\") pod \"kserve-controller-manager-fcc9dd4db-4t79k\" (UID: \"2cde6c4f-b5ff-4e16-a860-77959ec50852\") " pod="kserve/kserve-controller-manager-fcc9dd4db-4t79k"
Jul 31 14:56:13.219794 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.219782    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2cde6c4f-b5ff-4e16-a860-77959ec50852-cert\") pod \"kserve-controller-manager-fcc9dd4db-4t79k\" (UID: \"2cde6c4f-b5ff-4e16-a860-77959ec50852\") " pod="kserve/kserve-controller-manager-fcc9dd4db-4t79k"
Jul 31 14:56:13.222467 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.222437    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2cde6c4f-b5ff-4e16-a860-77959ec50852-cert\") pod \"kserve-controller-manager-fcc9dd4db-4t79k\" (UID: \"2cde6c4f-b5ff-4e16-a860-77959ec50852\") " pod="kserve/kserve-controller-manager-fcc9dd4db-4t79k"
Jul 31 14:56:13.232794 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.232768    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh7cf\" (UniqueName: \"kubernetes.io/projected/2cde6c4f-b5ff-4e16-a860-77959ec50852-kube-api-access-hh7cf\") pod \"kserve-controller-manager-fcc9dd4db-4t79k\" (UID: \"2cde6c4f-b5ff-4e16-a860-77959ec50852\") " pod="kserve/kserve-controller-manager-fcc9dd4db-4t79k"
Jul 31 14:56:13.304906 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.304866    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-fcc9dd4db-4t79k"
Jul 31 14:56:13.320422 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.320391    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1bba895-2fc9-41d1-a948-6f94043d3e90-cert\") pod \"e1bba895-2fc9-41d1-a948-6f94043d3e90\" (UID: \"e1bba895-2fc9-41d1-a948-6f94043d3e90\") "
Jul 31 14:56:13.320552 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.320440    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxv6j\" (UniqueName: \"kubernetes.io/projected/e1bba895-2fc9-41d1-a948-6f94043d3e90-kube-api-access-wxv6j\") pod \"e1bba895-2fc9-41d1-a948-6f94043d3e90\" (UID: \"e1bba895-2fc9-41d1-a948-6f94043d3e90\") "
Jul 31 14:56:13.322806 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.322781    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1bba895-2fc9-41d1-a948-6f94043d3e90-kube-api-access-wxv6j" (OuterVolumeSpecName: "kube-api-access-wxv6j") pod "e1bba895-2fc9-41d1-a948-6f94043d3e90" (UID: "e1bba895-2fc9-41d1-a948-6f94043d3e90"). InnerVolumeSpecName "kube-api-access-wxv6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:56:13.322806 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.322787    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1bba895-2fc9-41d1-a948-6f94043d3e90-cert" (OuterVolumeSpecName: "cert") pod "e1bba895-2fc9-41d1-a948-6f94043d3e90" (UID: "e1bba895-2fc9-41d1-a948-6f94043d3e90"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:56:13.420987 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.420958    2559 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e1bba895-2fc9-41d1-a948-6f94043d3e90-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 14:56:13.420987 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.420990    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxv6j\" (UniqueName: \"kubernetes.io/projected/e1bba895-2fc9-41d1-a948-6f94043d3e90-kube-api-access-wxv6j\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 14:56:13.447983 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.447945    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-fcc9dd4db-4t79k"]
Jul 31 14:56:13.448372 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:56:13.448346    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cde6c4f_b5ff_4e16_a860_77959ec50852.slice/crio-2a5ed9c90fe8d596b55d766cdab4ca0c9c9e3f3f8fdcf3bf8a5a87da3efe7db2 WatchSource:0}: Error finding container 2a5ed9c90fe8d596b55d766cdab4ca0c9c9e3f3f8fdcf3bf8a5a87da3efe7db2: Status 404 returned error can't find the container with id 2a5ed9c90fe8d596b55d766cdab4ca0c9c9e3f3f8fdcf3bf8a5a87da3efe7db2
Jul 31 14:56:13.913676 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.913639    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-fcc9dd4db-4t79k" event={"ID":"2cde6c4f-b5ff-4e16-a860-77959ec50852","Type":"ContainerStarted","Data":"1c912a3c990a3c6be1c15a2fa10795395fa34e2dfa58388f67eaa68cec9ea363"}
Jul 31 14:56:13.913844 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.913698    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-fcc9dd4db-4t79k" event={"ID":"2cde6c4f-b5ff-4e16-a860-77959ec50852","Type":"ContainerStarted","Data":"2a5ed9c90fe8d596b55d766cdab4ca0c9c9e3f3f8fdcf3bf8a5a87da3efe7db2"}
Jul 31 14:56:13.913922 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.913852    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-fcc9dd4db-4t79k"
Jul 31 14:56:13.914993 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.914968    2559 generic.go:356] "Generic (PLEG): container finished" podID="e1bba895-2fc9-41d1-a948-6f94043d3e90" containerID="27b1530a69677282218502180278dfa1fa9662377ff98d7205ed2918c723bac8" exitCode=0
Jul 31 14:56:13.915132 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.915004    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-fcc9dd4db-48crh"
Jul 31 14:56:13.915132 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.915018    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-fcc9dd4db-48crh" event={"ID":"e1bba895-2fc9-41d1-a948-6f94043d3e90","Type":"ContainerDied","Data":"27b1530a69677282218502180278dfa1fa9662377ff98d7205ed2918c723bac8"}
Jul 31 14:56:13.915132 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.915051    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-fcc9dd4db-48crh" event={"ID":"e1bba895-2fc9-41d1-a948-6f94043d3e90","Type":"ContainerDied","Data":"f542e72e36df992fbf6ce86cf3ab12fa4fb152655d1b3b8f1c11884903e6b040"}
Jul 31 14:56:13.915132 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.915072    2559 scope.go:117] "RemoveContainer" containerID="27b1530a69677282218502180278dfa1fa9662377ff98d7205ed2918c723bac8"
Jul 31 14:56:13.924503 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.924481    2559 scope.go:117] "RemoveContainer" containerID="27b1530a69677282218502180278dfa1fa9662377ff98d7205ed2918c723bac8"
Jul 31 14:56:13.924828 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:56:13.924808    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27b1530a69677282218502180278dfa1fa9662377ff98d7205ed2918c723bac8\": container with ID starting with 27b1530a69677282218502180278dfa1fa9662377ff98d7205ed2918c723bac8 not found: ID does not exist" containerID="27b1530a69677282218502180278dfa1fa9662377ff98d7205ed2918c723bac8"
Jul 31 14:56:13.924905 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.924838    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27b1530a69677282218502180278dfa1fa9662377ff98d7205ed2918c723bac8"} err="failed to get container status \"27b1530a69677282218502180278dfa1fa9662377ff98d7205ed2918c723bac8\": rpc error: code = NotFound desc = could not find container \"27b1530a69677282218502180278dfa1fa9662377ff98d7205ed2918c723bac8\": container with ID starting with 27b1530a69677282218502180278dfa1fa9662377ff98d7205ed2918c723bac8 not found: ID does not exist"
Jul 31 14:56:13.962069 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.962022    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-fcc9dd4db-4t79k" podStartSLOduration=1.583916369 podStartE2EDuration="1.962004869s" totalImagesPullingTime="378.0885ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:56:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:56:13.936568122 +0000 UTC m=+470.248125743" watchObservedRunningTime="2026-07-31 14:56:13.962004869 +0000 UTC m=+470.273562530"
Jul 31 14:56:13.962396 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.962146    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-fcc9dd4db-48crh"]
Jul 31 14:56:13.968481 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:13.968455    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-fcc9dd4db-48crh"]
Jul 31 14:56:14.220847 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:14.220818    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1bba895-2fc9-41d1-a948-6f94043d3e90" path="/var/lib/kubelet/pods/e1bba895-2fc9-41d1-a948-6f94043d3e90/volumes"
Jul 31 14:56:44.923256 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:56:44.923226    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-fcc9dd4db-4t79k"
Jul 31 14:58:31.824141 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:58:31.824106    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-kt4jv"]
Jul 31 14:58:31.824629 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:58:31.824380    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1bba895-2fc9-41d1-a948-6f94043d3e90" containerName="manager"
Jul 31 14:58:31.824629 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:58:31.824391    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1bba895-2fc9-41d1-a948-6f94043d3e90" containerName="manager"
Jul 31 14:58:31.824629 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:58:31.824437    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1bba895-2fc9-41d1-a948-6f94043d3e90" containerName="manager"
Jul 31 14:58:31.826215 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:58:31.826195    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-kt4jv"
Jul 31 14:58:31.828816 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:58:31.828794    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Jul 31 14:58:31.828896 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:58:31.828795    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 31 14:58:31.834923 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:58:31.834894    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-kt4jv"]
Jul 31 14:58:31.898959 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:58:31.898923    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f79860f3-d477-422f-89f4-0b0c625f2c9f-data\") pod \"seaweedfs-tls-custom-8b44d767d-kt4jv\" (UID: \"f79860f3-d477-422f-89f4-0b0c625f2c9f\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-kt4jv"
Jul 31 14:58:31.899154 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:58:31.898964    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/f79860f3-d477-422f-89f4-0b0c625f2c9f-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-kt4jv\" (UID: \"f79860f3-d477-422f-89f4-0b0c625f2c9f\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-kt4jv"
Jul 31 14:58:31.899154 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:58:31.899015    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsgb7\" (UniqueName: \"kubernetes.io/projected/f79860f3-d477-422f-89f4-0b0c625f2c9f-kube-api-access-bsgb7\") pod \"seaweedfs-tls-custom-8b44d767d-kt4jv\" (UID: \"f79860f3-d477-422f-89f4-0b0c625f2c9f\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-kt4jv"
Jul 31 14:58:31.999795 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:58:31.999759    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f79860f3-d477-422f-89f4-0b0c625f2c9f-data\") pod \"seaweedfs-tls-custom-8b44d767d-kt4jv\" (UID: \"f79860f3-d477-422f-89f4-0b0c625f2c9f\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-kt4jv"
Jul 31 14:58:31.999967 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:58:31.999817    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/f79860f3-d477-422f-89f4-0b0c625f2c9f-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-kt4jv\" (UID: \"f79860f3-d477-422f-89f4-0b0c625f2c9f\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-kt4jv"
Jul 31 14:58:31.999967 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:58:31.999844    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsgb7\" (UniqueName: \"kubernetes.io/projected/f79860f3-d477-422f-89f4-0b0c625f2c9f-kube-api-access-bsgb7\") pod \"seaweedfs-tls-custom-8b44d767d-kt4jv\" (UID: \"f79860f3-d477-422f-89f4-0b0c625f2c9f\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-kt4jv"
Jul 31 14:58:32.000194 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:58:32.000170    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f79860f3-d477-422f-89f4-0b0c625f2c9f-data\") pod \"seaweedfs-tls-custom-8b44d767d-kt4jv\" (UID: \"f79860f3-d477-422f-89f4-0b0c625f2c9f\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-kt4jv"
Jul 31 14:58:32.002429 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:58:32.002405    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/f79860f3-d477-422f-89f4-0b0c625f2c9f-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-kt4jv\" (UID: \"f79860f3-d477-422f-89f4-0b0c625f2c9f\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-kt4jv"
Jul 31 14:58:32.009710 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:58:32.009685    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsgb7\" (UniqueName: \"kubernetes.io/projected/f79860f3-d477-422f-89f4-0b0c625f2c9f-kube-api-access-bsgb7\") pod \"seaweedfs-tls-custom-8b44d767d-kt4jv\" (UID: \"f79860f3-d477-422f-89f4-0b0c625f2c9f\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-kt4jv"
Jul 31 14:58:32.136278 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:58:32.136241    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-kt4jv"
Jul 31 14:58:32.267559 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:58:32.267526    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-kt4jv"]
Jul 31 14:58:32.270835 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:58:32.270808    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79860f3_d477_422f_89f4_0b0c625f2c9f.slice/crio-5a1d569bda9ef37513ff67a6a5e7cfb03e38918769d51c1c6935a953bf73244d WatchSource:0}: Error finding container 5a1d569bda9ef37513ff67a6a5e7cfb03e38918769d51c1c6935a953bf73244d: Status 404 returned error can't find the container with id 5a1d569bda9ef37513ff67a6a5e7cfb03e38918769d51c1c6935a953bf73244d
Jul 31 14:58:32.398300 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:58:32.398211    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-kt4jv" event={"ID":"f79860f3-d477-422f-89f4-0b0c625f2c9f","Type":"ContainerStarted","Data":"5a1d569bda9ef37513ff67a6a5e7cfb03e38918769d51c1c6935a953bf73244d"}
Jul 31 14:58:33.402914 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:58:33.402881    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-kt4jv" event={"ID":"f79860f3-d477-422f-89f4-0b0c625f2c9f","Type":"ContainerStarted","Data":"7a30f386f35cd25c4243eb31cd1327d2cdc29a54bd45f3f34a2981c77454e9fc"}
Jul 31 14:58:33.420435 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:58:33.420391    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-kt4jv" podStartSLOduration=2.182182542 podStartE2EDuration="2.420374467s" totalImagesPullingTime="238.191925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:58:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:58:33.419685935 +0000 UTC m=+609.731243560" watchObservedRunningTime="2026-07-31 14:58:33.420374467 +0000 UTC m=+609.731932090"
Jul 31 14:59:10.969404 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:10.969370    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"]
Jul 31 14:59:10.971628 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:10.971614    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 14:59:10.974091 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:10.974064    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 14:59:10.974240 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:10.974135    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 14:59:10.974240 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:10.974148    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dd67w\""
Jul 31 14:59:10.974418 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:10.974402    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 14:59:10.974978 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:10.974961    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 31 14:59:10.975067 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:10.975018    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 14:59:10.975067 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:10.974964    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 31 14:59:10.988600 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:10.988567    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"]
Jul 31 14:59:10.989630 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:10.989599    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4db34e85-40b4-4598-8083-e5d662e4e49d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-8488d4785f-pdrss\" (UID: \"4db34e85-40b4-4598-8083-e5d662e4e49d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 14:59:10.989776 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:10.989655    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4db34e85-40b4-4598-8083-e5d662e4e49d-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-8488d4785f-pdrss\" (UID: \"4db34e85-40b4-4598-8083-e5d662e4e49d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 14:59:10.989776 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:10.989704    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m85t\" (UniqueName: \"kubernetes.io/projected/4db34e85-40b4-4598-8083-e5d662e4e49d-kube-api-access-6m85t\") pod \"isvc-sklearn-batcher-predictor-8488d4785f-pdrss\" (UID: \"4db34e85-40b4-4598-8083-e5d662e4e49d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 14:59:10.989776 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:10.989758    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4db34e85-40b4-4598-8083-e5d662e4e49d-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-8488d4785f-pdrss\" (UID: \"4db34e85-40b4-4598-8083-e5d662e4e49d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 14:59:10.989921 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:10.989807    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4db34e85-40b4-4598-8083-e5d662e4e49d-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-8488d4785f-pdrss\" (UID: \"4db34e85-40b4-4598-8083-e5d662e4e49d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 14:59:11.091016 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:11.090977    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4db34e85-40b4-4598-8083-e5d662e4e49d-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-8488d4785f-pdrss\" (UID: \"4db34e85-40b4-4598-8083-e5d662e4e49d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 14:59:11.091195 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:11.091037    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4db34e85-40b4-4598-8083-e5d662e4e49d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-8488d4785f-pdrss\" (UID: \"4db34e85-40b4-4598-8083-e5d662e4e49d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 14:59:11.091195 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:11.091058    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4db34e85-40b4-4598-8083-e5d662e4e49d-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-8488d4785f-pdrss\" (UID: \"4db34e85-40b4-4598-8083-e5d662e4e49d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 14:59:11.091195 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:11.091121    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m85t\" (UniqueName: \"kubernetes.io/projected/4db34e85-40b4-4598-8083-e5d662e4e49d-kube-api-access-6m85t\") pod \"isvc-sklearn-batcher-predictor-8488d4785f-pdrss\" (UID: \"4db34e85-40b4-4598-8083-e5d662e4e49d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 14:59:11.091195 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:11.091154    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4db34e85-40b4-4598-8083-e5d662e4e49d-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-8488d4785f-pdrss\" (UID: \"4db34e85-40b4-4598-8083-e5d662e4e49d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 14:59:11.091398 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:59:11.091332    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-serving-cert: secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Jul 31 14:59:11.091454 ip-10-0-142-219 kubenswrapper[2559]: E0731 14:59:11.091421    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4db34e85-40b4-4598-8083-e5d662e4e49d-proxy-tls podName:4db34e85-40b4-4598-8083-e5d662e4e49d nodeName:}" failed. No retries permitted until 2026-07-31 14:59:11.591398548 +0000 UTC m=+647.902956154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4db34e85-40b4-4598-8083-e5d662e4e49d-proxy-tls") pod "isvc-sklearn-batcher-predictor-8488d4785f-pdrss" (UID: "4db34e85-40b4-4598-8083-e5d662e4e49d") : secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Jul 31 14:59:11.091590 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:11.091569    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4db34e85-40b4-4598-8083-e5d662e4e49d-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-8488d4785f-pdrss\" (UID: \"4db34e85-40b4-4598-8083-e5d662e4e49d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 14:59:11.091812 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:11.091790    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4db34e85-40b4-4598-8083-e5d662e4e49d-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-8488d4785f-pdrss\" (UID: \"4db34e85-40b4-4598-8083-e5d662e4e49d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 14:59:11.091890 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:11.091795    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4db34e85-40b4-4598-8083-e5d662e4e49d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-8488d4785f-pdrss\" (UID: \"4db34e85-40b4-4598-8083-e5d662e4e49d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 14:59:11.102581 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:11.102544    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m85t\" (UniqueName: \"kubernetes.io/projected/4db34e85-40b4-4598-8083-e5d662e4e49d-kube-api-access-6m85t\") pod \"isvc-sklearn-batcher-predictor-8488d4785f-pdrss\" (UID: \"4db34e85-40b4-4598-8083-e5d662e4e49d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 14:59:11.595358 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:11.595307    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4db34e85-40b4-4598-8083-e5d662e4e49d-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-8488d4785f-pdrss\" (UID: \"4db34e85-40b4-4598-8083-e5d662e4e49d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 14:59:11.597859 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:11.597830    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4db34e85-40b4-4598-8083-e5d662e4e49d-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-8488d4785f-pdrss\" (UID: \"4db34e85-40b4-4598-8083-e5d662e4e49d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 14:59:11.882292 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:11.882211    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 14:59:12.040261 ip-10-0-142-219 kubenswrapper[2559]: W0731 14:59:12.040228    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db34e85_40b4_4598_8083_e5d662e4e49d.slice/crio-db6e33a2c1a589ab1e0bafa544d7fe3d579d9c31947f415ef75f2d834a2faf2f WatchSource:0}: Error finding container db6e33a2c1a589ab1e0bafa544d7fe3d579d9c31947f415ef75f2d834a2faf2f: Status 404 returned error can't find the container with id db6e33a2c1a589ab1e0bafa544d7fe3d579d9c31947f415ef75f2d834a2faf2f
Jul 31 14:59:12.048010 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:12.047985    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"]
Jul 31 14:59:12.539132 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:12.539092    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" event={"ID":"4db34e85-40b4-4598-8083-e5d662e4e49d","Type":"ContainerStarted","Data":"db6e33a2c1a589ab1e0bafa544d7fe3d579d9c31947f415ef75f2d834a2faf2f"}
Jul 31 14:59:15.552341 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:15.552249    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" event={"ID":"4db34e85-40b4-4598-8083-e5d662e4e49d","Type":"ContainerStarted","Data":"ca83be3bd245992d0e159253162bd939ef71280be97169e90cf43eb2865878ea"}
Jul 31 14:59:19.567954 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:19.567863    2559 generic.go:356] "Generic (PLEG): container finished" podID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerID="ca83be3bd245992d0e159253162bd939ef71280be97169e90cf43eb2865878ea" exitCode=0
Jul 31 14:59:19.567954 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:19.567930    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" event={"ID":"4db34e85-40b4-4598-8083-e5d662e4e49d","Type":"ContainerDied","Data":"ca83be3bd245992d0e159253162bd939ef71280be97169e90cf43eb2865878ea"}
Jul 31 14:59:32.634622 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:32.634536    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" event={"ID":"4db34e85-40b4-4598-8083-e5d662e4e49d","Type":"ContainerStarted","Data":"e219d5304a0a07649bc4ff980f8bfc74076edacdadab64df15098f9b6b04f202"}
Jul 31 14:59:35.650940 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:35.650901    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" event={"ID":"4db34e85-40b4-4598-8083-e5d662e4e49d","Type":"ContainerStarted","Data":"662cf304107ac9801974e1f423e68e8b3f25c9d414cc92b1aa4765c3cc8c2ca3"}
Jul 31 14:59:37.661967 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:37.661933    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" event={"ID":"4db34e85-40b4-4598-8083-e5d662e4e49d","Type":"ContainerStarted","Data":"d9c768a2a13d90f0ed541335cfd126eabc88c778cfffd621a857a99428ec867a"}
Jul 31 14:59:37.662414 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:37.662141    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 14:59:37.662414 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:37.662214    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 14:59:37.662414 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:37.662256    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 14:59:37.663612 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:37.663586    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 31 14:59:37.664231 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:37.664201    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 14:59:37.687023 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:37.686973    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podStartSLOduration=3.02076837 podStartE2EDuration="27.686959165s" totalImagesPullingTime="20.629226615s" totalInitContainerRuntime="4.03696418s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:59:10 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:59:37.686427646 +0000 UTC m=+673.997985278" watchObservedRunningTime="2026-07-31 14:59:37.686959165 +0000 UTC m=+673.998516786"
Jul 31 14:59:38.666070 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:38.666025    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 31 14:59:38.666469 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:38.666372    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 14:59:43.670136 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:43.670105    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 14:59:43.670708 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:43.670674    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 31 14:59:43.670941 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:43.670917    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 14:59:53.671267 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:53.671188    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 31 14:59:53.671801 ip-10-0-142-219 kubenswrapper[2559]: I0731 14:59:53.671719    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:00:03.670573 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:03.670530    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 31 15:00:03.671097 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:03.670978    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:00:13.670635 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:13.670583    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 31 15:00:13.671212 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:13.671000    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:00:23.671051 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:23.670936    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 31 15:00:23.671628 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:23.671304    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:00:33.671071 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:33.670972    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 31 15:00:33.671529 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:33.671434    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:00:43.670910 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:43.670873    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 31 15:00:43.671500 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:43.671461    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:00:47.216939 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:47.216903    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 15:00:47.217414 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:47.216990    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 15:00:55.980100 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:55.980057    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"]
Jul 31 15:00:55.982991 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:55.982955    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"
Jul 31 15:00:55.986323 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:55.986303    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 31 15:00:55.986441 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:55.986338    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 31 15:00:56.001766 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:56.001720    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"]
Jul 31 15:00:56.019453 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:56.019414    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"]
Jul 31 15:00:56.019719 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:56.019684    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="kserve-container" containerID="cri-o://e219d5304a0a07649bc4ff980f8bfc74076edacdadab64df15098f9b6b04f202" gracePeriod=30
Jul 31 15:00:56.019884 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:56.019764    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="agent" containerID="cri-o://d9c768a2a13d90f0ed541335cfd126eabc88c778cfffd621a857a99428ec867a" gracePeriod=30
Jul 31 15:00:56.019951 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:56.019772    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="kube-rbac-proxy" containerID="cri-o://662cf304107ac9801974e1f423e68e8b3f25c9d414cc92b1aa4765c3cc8c2ca3" gracePeriod=30
Jul 31 15:00:56.133403 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:56.133369    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4qj4\" (UniqueName: \"kubernetes.io/projected/757a6c4f-ad9a-420d-b233-449c16bc1784-kube-api-access-j4qj4\") pod \"isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt\" (UID: \"757a6c4f-ad9a-420d-b233-449c16bc1784\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"
Jul 31 15:00:56.133566 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:56.133434    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/757a6c4f-ad9a-420d-b233-449c16bc1784-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt\" (UID: \"757a6c4f-ad9a-420d-b233-449c16bc1784\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"
Jul 31 15:00:56.133566 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:56.133495    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/757a6c4f-ad9a-420d-b233-449c16bc1784-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt\" (UID: \"757a6c4f-ad9a-420d-b233-449c16bc1784\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"
Jul 31 15:00:56.133566 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:56.133541    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/757a6c4f-ad9a-420d-b233-449c16bc1784-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt\" (UID: \"757a6c4f-ad9a-420d-b233-449c16bc1784\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"
Jul 31 15:00:56.133682 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:56.133584    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/757a6c4f-ad9a-420d-b233-449c16bc1784-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt\" (UID: \"757a6c4f-ad9a-420d-b233-449c16bc1784\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"
Jul 31 15:00:56.234359 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:56.234265    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/757a6c4f-ad9a-420d-b233-449c16bc1784-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt\" (UID: \"757a6c4f-ad9a-420d-b233-449c16bc1784\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"
Jul 31 15:00:56.234359 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:56.234350    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4qj4\" (UniqueName: \"kubernetes.io/projected/757a6c4f-ad9a-420d-b233-449c16bc1784-kube-api-access-j4qj4\") pod \"isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt\" (UID: \"757a6c4f-ad9a-420d-b233-449c16bc1784\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"
Jul 31 15:00:56.234596 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:56.234411    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/757a6c4f-ad9a-420d-b233-449c16bc1784-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt\" (UID: \"757a6c4f-ad9a-420d-b233-449c16bc1784\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"
Jul 31 15:00:56.234596 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:56.234489    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/757a6c4f-ad9a-420d-b233-449c16bc1784-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt\" (UID: \"757a6c4f-ad9a-420d-b233-449c16bc1784\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"
Jul 31 15:00:56.234596 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:56.234547    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/757a6c4f-ad9a-420d-b233-449c16bc1784-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt\" (UID: \"757a6c4f-ad9a-420d-b233-449c16bc1784\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"
Jul 31 15:00:56.234838 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:56.234812    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/757a6c4f-ad9a-420d-b233-449c16bc1784-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt\" (UID: \"757a6c4f-ad9a-420d-b233-449c16bc1784\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"
Jul 31 15:00:56.235187 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:56.235152    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/757a6c4f-ad9a-420d-b233-449c16bc1784-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt\" (UID: \"757a6c4f-ad9a-420d-b233-449c16bc1784\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"
Jul 31 15:00:56.235423 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:56.235231    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/757a6c4f-ad9a-420d-b233-449c16bc1784-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt\" (UID: \"757a6c4f-ad9a-420d-b233-449c16bc1784\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"
Jul 31 15:00:56.237188 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:56.237168    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/757a6c4f-ad9a-420d-b233-449c16bc1784-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt\" (UID: \"757a6c4f-ad9a-420d-b233-449c16bc1784\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"
Jul 31 15:00:56.248191 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:56.248166    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4qj4\" (UniqueName: \"kubernetes.io/projected/757a6c4f-ad9a-420d-b233-449c16bc1784-kube-api-access-j4qj4\") pod \"isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt\" (UID: \"757a6c4f-ad9a-420d-b233-449c16bc1784\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"
Jul 31 15:00:56.293767 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:56.293707    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"
Jul 31 15:00:56.433423 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:56.433381    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"]
Jul 31 15:00:56.436468 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:00:56.436443    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod757a6c4f_ad9a_420d_b233_449c16bc1784.slice/crio-eeb34f85d82e8d7456505eda9155be3adb93ca6f76701b3bbab04e2fc9328859 WatchSource:0}: Error finding container eeb34f85d82e8d7456505eda9155be3adb93ca6f76701b3bbab04e2fc9328859: Status 404 returned error can't find the container with id eeb34f85d82e8d7456505eda9155be3adb93ca6f76701b3bbab04e2fc9328859
Jul 31 15:00:56.968022 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:56.967981    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" event={"ID":"757a6c4f-ad9a-420d-b233-449c16bc1784","Type":"ContainerStarted","Data":"de5b6a5a6435fe464c03af28c8ff42c659f647f361486497c403745f7b3c1ce0"}
Jul 31 15:00:56.968218 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:56.968057    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" event={"ID":"757a6c4f-ad9a-420d-b233-449c16bc1784","Type":"ContainerStarted","Data":"eeb34f85d82e8d7456505eda9155be3adb93ca6f76701b3bbab04e2fc9328859"}
Jul 31 15:00:56.970849 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:56.970825    2559 generic.go:356] "Generic (PLEG): container finished" podID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerID="662cf304107ac9801974e1f423e68e8b3f25c9d414cc92b1aa4765c3cc8c2ca3" exitCode=2
Jul 31 15:00:56.970967 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:56.970900    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" event={"ID":"4db34e85-40b4-4598-8083-e5d662e4e49d","Type":"ContainerDied","Data":"662cf304107ac9801974e1f423e68e8b3f25c9d414cc92b1aa4765c3cc8c2ca3"}
Jul 31 15:00:57.216517 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:57.216472    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 31 15:00:57.216957 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:57.216762    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:00:58.666713 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:00:58.666671    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 31 15:01:00.989367 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:00.989330    2559 generic.go:356] "Generic (PLEG): container finished" podID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerID="e219d5304a0a07649bc4ff980f8bfc74076edacdadab64df15098f9b6b04f202" exitCode=0
Jul 31 15:01:00.989787 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:00.989403    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" event={"ID":"4db34e85-40b4-4598-8083-e5d662e4e49d","Type":"ContainerDied","Data":"e219d5304a0a07649bc4ff980f8bfc74076edacdadab64df15098f9b6b04f202"}
Jul 31 15:01:00.991077 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:00.991057    2559 generic.go:356] "Generic (PLEG): container finished" podID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerID="de5b6a5a6435fe464c03af28c8ff42c659f647f361486497c403745f7b3c1ce0" exitCode=0
Jul 31 15:01:00.991194 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:00.991104    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" event={"ID":"757a6c4f-ad9a-420d-b233-449c16bc1784","Type":"ContainerDied","Data":"de5b6a5a6435fe464c03af28c8ff42c659f647f361486497c403745f7b3c1ce0"}
Jul 31 15:01:01.103444 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:01.103413    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"]
Jul 31 15:01:01.109066 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:01.109043    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"
Jul 31 15:01:01.111701 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:01.111655    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-10a884-predictor-serving-cert\""
Jul 31 15:01:01.111701 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:01.111687    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-10a884-kube-rbac-proxy-sar-config\""
Jul 31 15:01:01.117608 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:01.117550    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"]
Jul 31 15:01:01.274846 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:01.274809    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c79515a4-af1a-4622-be1c-8ed4d13e3302-cabundle-cert\") pod \"isvc-primary-10a884-predictor-565495c8c8-qvvlx\" (UID: \"c79515a4-af1a-4622-be1c-8ed4d13e3302\") " pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"
Jul 31 15:01:01.275033 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:01.274876    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c79515a4-af1a-4622-be1c-8ed4d13e3302-kube-rbac-proxy-sar-config\") pod \"isvc-primary-10a884-predictor-565495c8c8-qvvlx\" (UID: \"c79515a4-af1a-4622-be1c-8ed4d13e3302\") " pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"
Jul 31 15:01:01.275033 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:01.274905    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztgdg\" (UniqueName: \"kubernetes.io/projected/c79515a4-af1a-4622-be1c-8ed4d13e3302-kube-api-access-ztgdg\") pod \"isvc-primary-10a884-predictor-565495c8c8-qvvlx\" (UID: \"c79515a4-af1a-4622-be1c-8ed4d13e3302\") " pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"
Jul 31 15:01:01.275033 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:01.274961    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c79515a4-af1a-4622-be1c-8ed4d13e3302-kserve-provision-location\") pod \"isvc-primary-10a884-predictor-565495c8c8-qvvlx\" (UID: \"c79515a4-af1a-4622-be1c-8ed4d13e3302\") " pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"
Jul 31 15:01:01.275033 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:01.275004    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c79515a4-af1a-4622-be1c-8ed4d13e3302-proxy-tls\") pod \"isvc-primary-10a884-predictor-565495c8c8-qvvlx\" (UID: \"c79515a4-af1a-4622-be1c-8ed4d13e3302\") " pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"
Jul 31 15:01:01.378080 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:01.376914    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c79515a4-af1a-4622-be1c-8ed4d13e3302-kserve-provision-location\") pod \"isvc-primary-10a884-predictor-565495c8c8-qvvlx\" (UID: \"c79515a4-af1a-4622-be1c-8ed4d13e3302\") " pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"
Jul 31 15:01:01.378080 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:01.376997    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c79515a4-af1a-4622-be1c-8ed4d13e3302-proxy-tls\") pod \"isvc-primary-10a884-predictor-565495c8c8-qvvlx\" (UID: \"c79515a4-af1a-4622-be1c-8ed4d13e3302\") " pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"
Jul 31 15:01:01.378080 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:01:01.377132    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-10a884-predictor-serving-cert: secret "isvc-primary-10a884-predictor-serving-cert" not found
Jul 31 15:01:01.378080 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:01:01.377222    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c79515a4-af1a-4622-be1c-8ed4d13e3302-proxy-tls podName:c79515a4-af1a-4622-be1c-8ed4d13e3302 nodeName:}" failed. No retries permitted until 2026-07-31 15:01:01.877189857 +0000 UTC m=+758.188747468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c79515a4-af1a-4622-be1c-8ed4d13e3302-proxy-tls") pod "isvc-primary-10a884-predictor-565495c8c8-qvvlx" (UID: "c79515a4-af1a-4622-be1c-8ed4d13e3302") : secret "isvc-primary-10a884-predictor-serving-cert" not found
Jul 31 15:01:01.378080 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:01.377351    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c79515a4-af1a-4622-be1c-8ed4d13e3302-cabundle-cert\") pod \"isvc-primary-10a884-predictor-565495c8c8-qvvlx\" (UID: \"c79515a4-af1a-4622-be1c-8ed4d13e3302\") " pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"
Jul 31 15:01:01.378080 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:01.377490    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c79515a4-af1a-4622-be1c-8ed4d13e3302-kube-rbac-proxy-sar-config\") pod \"isvc-primary-10a884-predictor-565495c8c8-qvvlx\" (UID: \"c79515a4-af1a-4622-be1c-8ed4d13e3302\") " pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"
Jul 31 15:01:01.378468 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:01.378411    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c79515a4-af1a-4622-be1c-8ed4d13e3302-cabundle-cert\") pod \"isvc-primary-10a884-predictor-565495c8c8-qvvlx\" (UID: \"c79515a4-af1a-4622-be1c-8ed4d13e3302\") " pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"
Jul 31 15:01:01.381966 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:01.378632    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztgdg\" (UniqueName: \"kubernetes.io/projected/c79515a4-af1a-4622-be1c-8ed4d13e3302-kube-api-access-ztgdg\") pod \"isvc-primary-10a884-predictor-565495c8c8-qvvlx\" (UID: \"c79515a4-af1a-4622-be1c-8ed4d13e3302\") " pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"
Jul 31 15:01:01.381966 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:01.378846    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c79515a4-af1a-4622-be1c-8ed4d13e3302-kserve-provision-location\") pod \"isvc-primary-10a884-predictor-565495c8c8-qvvlx\" (UID: \"c79515a4-af1a-4622-be1c-8ed4d13e3302\") " pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"
Jul 31 15:01:01.381966 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:01.378993    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c79515a4-af1a-4622-be1c-8ed4d13e3302-kube-rbac-proxy-sar-config\") pod \"isvc-primary-10a884-predictor-565495c8c8-qvvlx\" (UID: \"c79515a4-af1a-4622-be1c-8ed4d13e3302\") " pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"
Jul 31 15:01:01.387999 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:01.387972    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztgdg\" (UniqueName: \"kubernetes.io/projected/c79515a4-af1a-4622-be1c-8ed4d13e3302-kube-api-access-ztgdg\") pod \"isvc-primary-10a884-predictor-565495c8c8-qvvlx\" (UID: \"c79515a4-af1a-4622-be1c-8ed4d13e3302\") " pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"
Jul 31 15:01:01.882982 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:01.882939    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c79515a4-af1a-4622-be1c-8ed4d13e3302-proxy-tls\") pod \"isvc-primary-10a884-predictor-565495c8c8-qvvlx\" (UID: \"c79515a4-af1a-4622-be1c-8ed4d13e3302\") " pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"
Jul 31 15:01:01.885600 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:01.885571    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c79515a4-af1a-4622-be1c-8ed4d13e3302-proxy-tls\") pod \"isvc-primary-10a884-predictor-565495c8c8-qvvlx\" (UID: \"c79515a4-af1a-4622-be1c-8ed4d13e3302\") " pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"
Jul 31 15:01:01.997469 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:01.997432    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" event={"ID":"757a6c4f-ad9a-420d-b233-449c16bc1784","Type":"ContainerStarted","Data":"46bebb7b027a41f3ff2ae26debb7e454f2633a7fd28b49eeb10eb38b09f44545"}
Jul 31 15:01:01.997883 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:01.997528    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" event={"ID":"757a6c4f-ad9a-420d-b233-449c16bc1784","Type":"ContainerStarted","Data":"5b45920908bc6be71bbb7a264febddbc212b1c3192fdb25cde720324f7d4fb57"}
Jul 31 15:01:01.997883 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:01.997553    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" event={"ID":"757a6c4f-ad9a-420d-b233-449c16bc1784","Type":"ContainerStarted","Data":"780888eb275bf6b249cb33633a3827049b356bf8a2ee983aeb2089831d18605a"}
Jul 31 15:01:01.997986 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:01.997899    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"
Jul 31 15:01:01.998069 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:01.998051    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"
Jul 31 15:01:01.999514 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:01.999487    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:5000: connect: connection refused"
Jul 31 15:01:02.023082 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:02.023044    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"
Jul 31 15:01:02.028216 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:02.028159    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podStartSLOduration=3.173987631 podStartE2EDuration="7.028142091s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.85415446s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:01:02.026929722 +0000 UTC m=+758.338487344" watchObservedRunningTime="2026-07-31 15:01:02.028142091 +0000 UTC m=+758.339699712"
Jul 31 15:01:02.184188 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:02.184158    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"]
Jul 31 15:01:02.186720 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:01:02.186692    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc79515a4_af1a_4622_be1c_8ed4d13e3302.slice/crio-b1c67e018409e1cc5e76f53c55b14d55e49b8dc075ed26140898920dc01d08a9 WatchSource:0}: Error finding container b1c67e018409e1cc5e76f53c55b14d55e49b8dc075ed26140898920dc01d08a9: Status 404 returned error can't find the container with id b1c67e018409e1cc5e76f53c55b14d55e49b8dc075ed26140898920dc01d08a9
Jul 31 15:01:03.003072 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:03.003029    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx" event={"ID":"c79515a4-af1a-4622-be1c-8ed4d13e3302","Type":"ContainerStarted","Data":"ee7b17723fa5a6766cb331ebfe932dc2edf75be7f3eb7bc92d4ccddb8c3ab06e"}
Jul 31 15:01:03.003501 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:03.003104    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx" event={"ID":"c79515a4-af1a-4622-be1c-8ed4d13e3302","Type":"ContainerStarted","Data":"b1c67e018409e1cc5e76f53c55b14d55e49b8dc075ed26140898920dc01d08a9"}
Jul 31 15:01:03.003501 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:03.003452    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"
Jul 31 15:01:03.003501 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:03.003471    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:5000: connect: connection refused"
Jul 31 15:01:03.004765 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:03.004719    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:01:03.667038 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:03.666986    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 31 15:01:04.007872 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:04.007770    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:5000: connect: connection refused"
Jul 31 15:01:04.008284 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:04.008258    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:01:07.021164 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:07.021126    2559 generic.go:356] "Generic (PLEG): container finished" podID="c79515a4-af1a-4622-be1c-8ed4d13e3302" containerID="ee7b17723fa5a6766cb331ebfe932dc2edf75be7f3eb7bc92d4ccddb8c3ab06e" exitCode=0
Jul 31 15:01:07.021555 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:07.021191    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx" event={"ID":"c79515a4-af1a-4622-be1c-8ed4d13e3302","Type":"ContainerDied","Data":"ee7b17723fa5a6766cb331ebfe932dc2edf75be7f3eb7bc92d4ccddb8c3ab06e"}
Jul 31 15:01:07.216449 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:07.216406    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 31 15:01:07.216804 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:07.216784    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:01:08.027282 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:08.027243    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx" event={"ID":"c79515a4-af1a-4622-be1c-8ed4d13e3302","Type":"ContainerStarted","Data":"b4d29867c8d702f52f75468bf4801b7869bed6a2593399ad44697e69afd00c50"}
Jul 31 15:01:08.027660 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:08.027313    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx" event={"ID":"c79515a4-af1a-4622-be1c-8ed4d13e3302","Type":"ContainerStarted","Data":"a68c6e57662858e0068d7ed522bfbf91f7152255938907708384aa66e6075480"}
Jul 31 15:01:08.027660 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:08.027575    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"
Jul 31 15:01:08.052450 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:08.052401    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx" podStartSLOduration=3.097048341 podStartE2EDuration="7.052381645s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.955333304s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:01:08.051696174 +0000 UTC m=+764.363253810" watchObservedRunningTime="2026-07-31 15:01:08.052381645 +0000 UTC m=+764.363939269"
Jul 31 15:01:08.666509 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:08.666467    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 31 15:01:08.666701 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:08.666617    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 15:01:09.012277 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:09.012198    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"
Jul 31 15:01:09.012906 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:09.012875    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:5000: connect: connection refused"
Jul 31 15:01:09.013231 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:09.013208    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:01:09.031791 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:09.031765    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"
Jul 31 15:01:09.032996 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:09.032970    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx" podUID="c79515a4-af1a-4622-be1c-8ed4d13e3302" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 31 15:01:10.035642 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:10.035600    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx" podUID="c79515a4-af1a-4622-be1c-8ed4d13e3302" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 31 15:01:13.666606 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:13.666547    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 31 15:01:15.040173 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:15.040146    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"
Jul 31 15:01:15.040635 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:15.040610    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx" podUID="c79515a4-af1a-4622-be1c-8ed4d13e3302" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 31 15:01:17.216462 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:17.216411    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 31 15:01:17.216921 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:17.216580    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 15:01:17.216921 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:17.216683    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:01:17.216921 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:17.216860    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 15:01:18.666566 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:18.666520    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 31 15:01:19.013177 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:19.013073    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:5000: connect: connection refused"
Jul 31 15:01:19.013556 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:19.013531    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:01:23.666307 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:23.666264    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 31 15:01:25.040787 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:25.040751    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx" podUID="c79515a4-af1a-4622-be1c-8ed4d13e3302" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 31 15:01:26.108769 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:26.108742    2559 generic.go:356] "Generic (PLEG): container finished" podID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerID="d9c768a2a13d90f0ed541335cfd126eabc88c778cfffd621a857a99428ec867a" exitCode=0
Jul 31 15:01:26.109135 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:26.108770    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" event={"ID":"4db34e85-40b4-4598-8083-e5d662e4e49d","Type":"ContainerDied","Data":"d9c768a2a13d90f0ed541335cfd126eabc88c778cfffd621a857a99428ec867a"}
Jul 31 15:01:26.166620 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:26.166586    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 15:01:26.185712 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:26.185681    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4db34e85-40b4-4598-8083-e5d662e4e49d-kube-rbac-proxy-sar-config\") pod \"4db34e85-40b4-4598-8083-e5d662e4e49d\" (UID: \"4db34e85-40b4-4598-8083-e5d662e4e49d\") "
Jul 31 15:01:26.185866 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:26.185748    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4db34e85-40b4-4598-8083-e5d662e4e49d-kserve-provision-location\") pod \"4db34e85-40b4-4598-8083-e5d662e4e49d\" (UID: \"4db34e85-40b4-4598-8083-e5d662e4e49d\") "
Jul 31 15:01:26.185866 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:26.185767    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6m85t\" (UniqueName: \"kubernetes.io/projected/4db34e85-40b4-4598-8083-e5d662e4e49d-kube-api-access-6m85t\") pod \"4db34e85-40b4-4598-8083-e5d662e4e49d\" (UID: \"4db34e85-40b4-4598-8083-e5d662e4e49d\") "
Jul 31 15:01:26.185866 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:26.185798    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4db34e85-40b4-4598-8083-e5d662e4e49d-proxy-tls\") pod \"4db34e85-40b4-4598-8083-e5d662e4e49d\" (UID: \"4db34e85-40b4-4598-8083-e5d662e4e49d\") "
Jul 31 15:01:26.185866 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:26.185818    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4db34e85-40b4-4598-8083-e5d662e4e49d-cabundle-cert\") pod \"4db34e85-40b4-4598-8083-e5d662e4e49d\" (UID: \"4db34e85-40b4-4598-8083-e5d662e4e49d\") "
Jul 31 15:01:26.186097 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:26.186060    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db34e85-40b4-4598-8083-e5d662e4e49d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4db34e85-40b4-4598-8083-e5d662e4e49d" (UID: "4db34e85-40b4-4598-8083-e5d662e4e49d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:01:26.186178 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:26.186099    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db34e85-40b4-4598-8083-e5d662e4e49d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4db34e85-40b4-4598-8083-e5d662e4e49d" (UID: "4db34e85-40b4-4598-8083-e5d662e4e49d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:01:26.186280 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:26.186257    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db34e85-40b4-4598-8083-e5d662e4e49d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4db34e85-40b4-4598-8083-e5d662e4e49d" (UID: "4db34e85-40b4-4598-8083-e5d662e4e49d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:01:26.188165 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:26.188138    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db34e85-40b4-4598-8083-e5d662e4e49d-kube-api-access-6m85t" (OuterVolumeSpecName: "kube-api-access-6m85t") pod "4db34e85-40b4-4598-8083-e5d662e4e49d" (UID: "4db34e85-40b4-4598-8083-e5d662e4e49d"). InnerVolumeSpecName "kube-api-access-6m85t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:01:26.188253 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:26.188162    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4db34e85-40b4-4598-8083-e5d662e4e49d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4db34e85-40b4-4598-8083-e5d662e4e49d" (UID: "4db34e85-40b4-4598-8083-e5d662e4e49d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:01:26.286583 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:26.286500    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4db34e85-40b4-4598-8083-e5d662e4e49d-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:01:26.286583 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:26.286535    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m85t\" (UniqueName: \"kubernetes.io/projected/4db34e85-40b4-4598-8083-e5d662e4e49d-kube-api-access-6m85t\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:01:26.286583 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:26.286545    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4db34e85-40b4-4598-8083-e5d662e4e49d-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:01:26.286583 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:26.286555    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4db34e85-40b4-4598-8083-e5d662e4e49d-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:01:26.286583 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:26.286563    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4db34e85-40b4-4598-8083-e5d662e4e49d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:01:27.114801 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:27.114768    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss" event={"ID":"4db34e85-40b4-4598-8083-e5d662e4e49d","Type":"ContainerDied","Data":"db6e33a2c1a589ab1e0bafa544d7fe3d579d9c31947f415ef75f2d834a2faf2f"}
Jul 31 15:01:27.114801 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:27.114794    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"
Jul 31 15:01:27.115421 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:27.114855    2559 scope.go:117] "RemoveContainer" containerID="d9c768a2a13d90f0ed541335cfd126eabc88c778cfffd621a857a99428ec867a"
Jul 31 15:01:27.123005 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:27.122984    2559 scope.go:117] "RemoveContainer" containerID="662cf304107ac9801974e1f423e68e8b3f25c9d414cc92b1aa4765c3cc8c2ca3"
Jul 31 15:01:27.130374 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:27.130354    2559 scope.go:117] "RemoveContainer" containerID="e219d5304a0a07649bc4ff980f8bfc74076edacdadab64df15098f9b6b04f202"
Jul 31 15:01:27.134202 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:27.134179    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"]
Jul 31 15:01:27.137686 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:27.137670    2559 scope.go:117] "RemoveContainer" containerID="ca83be3bd245992d0e159253162bd939ef71280be97169e90cf43eb2865878ea"
Jul 31 15:01:27.141198 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:27.141177    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-8488d4785f-pdrss"]
Jul 31 15:01:28.221218 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:28.221181    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" path="/var/lib/kubelet/pods/4db34e85-40b4-4598-8083-e5d662e4e49d/volumes"
Jul 31 15:01:29.012912 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:29.012871    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:5000: connect: connection refused"
Jul 31 15:01:29.013347 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:29.013323    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:01:35.040964 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:35.040925    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx" podUID="c79515a4-af1a-4622-be1c-8ed4d13e3302" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 31 15:01:39.013487 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:39.013433    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:5000: connect: connection refused"
Jul 31 15:01:39.013937 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:39.013905    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:01:45.041656 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:45.041580    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx" podUID="c79515a4-af1a-4622-be1c-8ed4d13e3302" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 31 15:01:49.013445 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:49.013402    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:5000: connect: connection refused"
Jul 31 15:01:49.013904 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:49.013868    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:01:55.040832 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:55.040795    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx" podUID="c79515a4-af1a-4622-be1c-8ed4d13e3302" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 31 15:01:59.013359 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:59.013320    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:5000: connect: connection refused"
Jul 31 15:01:59.013840 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:01:59.013780    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:02:05.040969 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:05.040923    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx" podUID="c79515a4-af1a-4622-be1c-8ed4d13e3302" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 31 15:02:09.013518 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:09.013471    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:5000: connect: connection refused"
Jul 31 15:02:09.013945 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:09.013917    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:02:15.040598 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:15.040555    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx" podUID="c79515a4-af1a-4622-be1c-8ed4d13e3302" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 31 15:02:19.013511 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:19.013479    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"
Jul 31 15:02:19.013965 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:19.013758    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"
Jul 31 15:02:21.217819 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:21.217791    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"
Jul 31 15:02:31.047430 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.047394    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"]
Jul 31 15:02:31.047963 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.047652    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="kserve-container" containerID="cri-o://780888eb275bf6b249cb33633a3827049b356bf8a2ee983aeb2089831d18605a" gracePeriod=30
Jul 31 15:02:31.047963 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.047677    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="agent" containerID="cri-o://46bebb7b027a41f3ff2ae26debb7e454f2633a7fd28b49eeb10eb38b09f44545" gracePeriod=30
Jul 31 15:02:31.047963 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.047684    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="kube-rbac-proxy" containerID="cri-o://5b45920908bc6be71bbb7a264febddbc212b1c3192fdb25cde720324f7d4fb57" gracePeriod=30
Jul 31 15:02:31.159554 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.159524    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q"]
Jul 31 15:02:31.162370 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.160268    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="storage-initializer"
Jul 31 15:02:31.162370 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.160292    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="storage-initializer"
Jul 31 15:02:31.162370 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.160315    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="agent"
Jul 31 15:02:31.162370 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.160324    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="agent"
Jul 31 15:02:31.162370 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.160365    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="kserve-container"
Jul 31 15:02:31.162370 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.160375    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="kserve-container"
Jul 31 15:02:31.162370 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.160414    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="kube-rbac-proxy"
Jul 31 15:02:31.162370 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.160425    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="kube-rbac-proxy"
Jul 31 15:02:31.162370 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.160635    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="kube-rbac-proxy"
Jul 31 15:02:31.162370 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.160648    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="agent"
Jul 31 15:02:31.162370 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.160669    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4db34e85-40b4-4598-8083-e5d662e4e49d" containerName="kserve-container"
Jul 31 15:02:31.167241 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.167220    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q"
Jul 31 15:02:31.169538 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.169518    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 31 15:02:31.169538 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.169529    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 31 15:02:31.173837 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.173815    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q"]
Jul 31 15:02:31.200372 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.200344    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srrzh\" (UniqueName: \"kubernetes.io/projected/b189726d-1b57-4ec4-876d-37850bb398d9-kube-api-access-srrzh\") pod \"message-dumper-predictor-6575bcfdfd-kl99q\" (UID: \"b189726d-1b57-4ec4-876d-37850bb398d9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q"
Jul 31 15:02:31.200521 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.200396    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b189726d-1b57-4ec4-876d-37850bb398d9-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-kl99q\" (UID: \"b189726d-1b57-4ec4-876d-37850bb398d9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q"
Jul 31 15:02:31.200521 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.200495    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b189726d-1b57-4ec4-876d-37850bb398d9-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-kl99q\" (UID: \"b189726d-1b57-4ec4-876d-37850bb398d9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q"
Jul 31 15:02:31.301767 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.301648    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b189726d-1b57-4ec4-876d-37850bb398d9-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-kl99q\" (UID: \"b189726d-1b57-4ec4-876d-37850bb398d9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q"
Jul 31 15:02:31.301767 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.301699    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srrzh\" (UniqueName: \"kubernetes.io/projected/b189726d-1b57-4ec4-876d-37850bb398d9-kube-api-access-srrzh\") pod \"message-dumper-predictor-6575bcfdfd-kl99q\" (UID: \"b189726d-1b57-4ec4-876d-37850bb398d9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q"
Jul 31 15:02:31.301959 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.301776    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b189726d-1b57-4ec4-876d-37850bb398d9-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-kl99q\" (UID: \"b189726d-1b57-4ec4-876d-37850bb398d9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q"
Jul 31 15:02:31.301959 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:02:31.301843    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
Jul 31 15:02:31.301959 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:02:31.301922    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b189726d-1b57-4ec4-876d-37850bb398d9-proxy-tls podName:b189726d-1b57-4ec4-876d-37850bb398d9 nodeName:}" failed. No retries permitted until 2026-07-31 15:02:31.801901636 +0000 UTC m=+848.113459241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b189726d-1b57-4ec4-876d-37850bb398d9-proxy-tls") pod "message-dumper-predictor-6575bcfdfd-kl99q" (UID: "b189726d-1b57-4ec4-876d-37850bb398d9") : secret "message-dumper-predictor-serving-cert" not found
Jul 31 15:02:31.302450 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.302432    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b189726d-1b57-4ec4-876d-37850bb398d9-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-kl99q\" (UID: \"b189726d-1b57-4ec4-876d-37850bb398d9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q"
Jul 31 15:02:31.314674 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.314644    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srrzh\" (UniqueName: \"kubernetes.io/projected/b189726d-1b57-4ec4-876d-37850bb398d9-kube-api-access-srrzh\") pod \"message-dumper-predictor-6575bcfdfd-kl99q\" (UID: \"b189726d-1b57-4ec4-876d-37850bb398d9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q"
Jul 31 15:02:31.362426 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.362391    2559 generic.go:356] "Generic (PLEG): container finished" podID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerID="5b45920908bc6be71bbb7a264febddbc212b1c3192fdb25cde720324f7d4fb57" exitCode=2
Jul 31 15:02:31.362594 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.362467    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" event={"ID":"757a6c4f-ad9a-420d-b233-449c16bc1784","Type":"ContainerDied","Data":"5b45920908bc6be71bbb7a264febddbc212b1c3192fdb25cde720324f7d4fb57"}
Jul 31 15:02:31.806310 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.806273    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b189726d-1b57-4ec4-876d-37850bb398d9-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-kl99q\" (UID: \"b189726d-1b57-4ec4-876d-37850bb398d9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q"
Jul 31 15:02:31.808893 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:31.808870    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b189726d-1b57-4ec4-876d-37850bb398d9-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-kl99q\" (UID: \"b189726d-1b57-4ec4-876d-37850bb398d9\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q"
Jul 31 15:02:32.078935 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:32.078843    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q"
Jul 31 15:02:32.210349 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:32.210312    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q"]
Jul 31 15:02:32.212786 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:02:32.212760    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb189726d_1b57_4ec4_876d_37850bb398d9.slice/crio-6c366197063cf0392510f830d353af256c1275203d645eac2b872e1b6c241ced WatchSource:0}: Error finding container 6c366197063cf0392510f830d353af256c1275203d645eac2b872e1b6c241ced: Status 404 returned error can't find the container with id 6c366197063cf0392510f830d353af256c1275203d645eac2b872e1b6c241ced
Jul 31 15:02:32.215130 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:32.215113    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 15:02:32.368052 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:32.368018    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q" event={"ID":"b189726d-1b57-4ec4-876d-37850bb398d9","Type":"ContainerStarted","Data":"6c366197063cf0392510f830d353af256c1275203d645eac2b872e1b6c241ced"}
Jul 31 15:02:34.008397 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:34.008359    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 31 15:02:34.378034 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:34.378000    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q" event={"ID":"b189726d-1b57-4ec4-876d-37850bb398d9","Type":"ContainerStarted","Data":"3c1c163e996b4750ecf493a4231b8e62723587341dffcda0aaed2a2c8de72faf"}
Jul 31 15:02:34.378228 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:34.378056    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q" event={"ID":"b189726d-1b57-4ec4-876d-37850bb398d9","Type":"ContainerStarted","Data":"38b1661ddfdb2d39e050c0ba049f1828af9aceb003de7269fbaa8614d48ccf16"}
Jul 31 15:02:34.378228 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:34.378149    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q"
Jul 31 15:02:34.398571 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:34.398522    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q" podStartSLOduration=2.163703768 podStartE2EDuration="3.398504676s" totalImagesPullingTime="1.234800908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:02:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:02:34.397775711 +0000 UTC m=+850.709333333" watchObservedRunningTime="2026-07-31 15:02:34.398504676 +0000 UTC m=+850.710062364"
Jul 31 15:02:35.382295 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:35.382256    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q"
Jul 31 15:02:35.383954 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:35.383929    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q"
Jul 31 15:02:36.393476 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:36.393442    2559 generic.go:356] "Generic (PLEG): container finished" podID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerID="780888eb275bf6b249cb33633a3827049b356bf8a2ee983aeb2089831d18605a" exitCode=0
Jul 31 15:02:36.393879 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:36.393514    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" event={"ID":"757a6c4f-ad9a-420d-b233-449c16bc1784","Type":"ContainerDied","Data":"780888eb275bf6b249cb33633a3827049b356bf8a2ee983aeb2089831d18605a"}
Jul 31 15:02:39.008716 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:39.008666    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 31 15:02:39.013077 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:39.013035    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:5000: connect: connection refused"
Jul 31 15:02:39.013420 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:39.013395    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:02:42.401875 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:42.401845    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q"
Jul 31 15:02:44.007876 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:44.007838    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 31 15:02:44.008269 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:44.007985    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"
Jul 31 15:02:46.292690 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:46.292653    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s"]
Jul 31 15:02:46.296232 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:46.296214    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s"
Jul 31 15:02:46.298986 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:46.298963    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-10a884-predictor-serving-cert\""
Jul 31 15:02:46.299101 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:46.299000    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-10a884-kube-rbac-proxy-sar-config\""
Jul 31 15:02:46.299101 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:46.299000    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-10a884\""
Jul 31 15:02:46.299101 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:46.299002    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-10a884-dockercfg-6wbhh\""
Jul 31 15:02:46.306802 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:46.306782    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s"]
Jul 31 15:02:46.416544 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:46.416505    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz89g\" (UniqueName: \"kubernetes.io/projected/d0424baf-32d5-4fea-820d-9f3043bf7ed3-kube-api-access-tz89g\") pod \"isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s\" (UID: \"d0424baf-32d5-4fea-820d-9f3043bf7ed3\") " pod="kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s"
Jul 31 15:02:46.416544 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:46.416546    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0424baf-32d5-4fea-820d-9f3043bf7ed3-kserve-provision-location\") pod \"isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s\" (UID: \"d0424baf-32d5-4fea-820d-9f3043bf7ed3\") " pod="kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s"
Jul 31 15:02:46.416798 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:46.416583    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0424baf-32d5-4fea-820d-9f3043bf7ed3-cabundle-cert\") pod \"isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s\" (UID: \"d0424baf-32d5-4fea-820d-9f3043bf7ed3\") " pod="kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s"
Jul 31 15:02:46.416798 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:46.416662    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0424baf-32d5-4fea-820d-9f3043bf7ed3-proxy-tls\") pod \"isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s\" (UID: \"d0424baf-32d5-4fea-820d-9f3043bf7ed3\") " pod="kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s"
Jul 31 15:02:46.416798 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:46.416691    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0424baf-32d5-4fea-820d-9f3043bf7ed3-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s\" (UID: \"d0424baf-32d5-4fea-820d-9f3043bf7ed3\") " pod="kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s"
Jul 31 15:02:46.517960 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:46.517930    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0424baf-32d5-4fea-820d-9f3043bf7ed3-proxy-tls\") pod \"isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s\" (UID: \"d0424baf-32d5-4fea-820d-9f3043bf7ed3\") " pod="kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s"
Jul 31 15:02:46.518119 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:46.517964    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0424baf-32d5-4fea-820d-9f3043bf7ed3-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s\" (UID: \"d0424baf-32d5-4fea-820d-9f3043bf7ed3\") " pod="kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s"
Jul 31 15:02:46.518119 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:46.518012    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz89g\" (UniqueName: \"kubernetes.io/projected/d0424baf-32d5-4fea-820d-9f3043bf7ed3-kube-api-access-tz89g\") pod \"isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s\" (UID: \"d0424baf-32d5-4fea-820d-9f3043bf7ed3\") " pod="kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s"
Jul 31 15:02:46.518119 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:46.518029    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0424baf-32d5-4fea-820d-9f3043bf7ed3-kserve-provision-location\") pod \"isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s\" (UID: \"d0424baf-32d5-4fea-820d-9f3043bf7ed3\") " pod="kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s"
Jul 31 15:02:46.518119 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:46.518052    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0424baf-32d5-4fea-820d-9f3043bf7ed3-cabundle-cert\") pod \"isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s\" (UID: \"d0424baf-32d5-4fea-820d-9f3043bf7ed3\") " pod="kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s"
Jul 31 15:02:46.518119 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:02:46.518091    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-10a884-predictor-serving-cert: secret "isvc-secondary-10a884-predictor-serving-cert" not found
Jul 31 15:02:46.518369 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:02:46.518181    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0424baf-32d5-4fea-820d-9f3043bf7ed3-proxy-tls podName:d0424baf-32d5-4fea-820d-9f3043bf7ed3 nodeName:}" failed. No retries permitted until 2026-07-31 15:02:47.01816078 +0000 UTC m=+863.329718379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d0424baf-32d5-4fea-820d-9f3043bf7ed3-proxy-tls") pod "isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s" (UID: "d0424baf-32d5-4fea-820d-9f3043bf7ed3") : secret "isvc-secondary-10a884-predictor-serving-cert" not found
Jul 31 15:02:46.518536 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:46.518518    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0424baf-32d5-4fea-820d-9f3043bf7ed3-kserve-provision-location\") pod \"isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s\" (UID: \"d0424baf-32d5-4fea-820d-9f3043bf7ed3\") " pod="kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s"
Jul 31 15:02:46.518685 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:46.518669    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0424baf-32d5-4fea-820d-9f3043bf7ed3-cabundle-cert\") pod \"isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s\" (UID: \"d0424baf-32d5-4fea-820d-9f3043bf7ed3\") " pod="kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s"
Jul 31 15:02:46.518756 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:46.518682    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0424baf-32d5-4fea-820d-9f3043bf7ed3-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s\" (UID: \"d0424baf-32d5-4fea-820d-9f3043bf7ed3\") " pod="kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s"
Jul 31 15:02:46.527194 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:46.527169    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz89g\" (UniqueName: \"kubernetes.io/projected/d0424baf-32d5-4fea-820d-9f3043bf7ed3-kube-api-access-tz89g\") pod \"isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s\" (UID: \"d0424baf-32d5-4fea-820d-9f3043bf7ed3\") " pod="kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s"
Jul 31 15:02:47.022975 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:47.022938    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0424baf-32d5-4fea-820d-9f3043bf7ed3-proxy-tls\") pod \"isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s\" (UID: \"d0424baf-32d5-4fea-820d-9f3043bf7ed3\") " pod="kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s"
Jul 31 15:02:47.025521 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:47.025494    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0424baf-32d5-4fea-820d-9f3043bf7ed3-proxy-tls\") pod \"isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s\" (UID: \"d0424baf-32d5-4fea-820d-9f3043bf7ed3\") " pod="kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s"
Jul 31 15:02:47.207554 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:47.207518    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s"
Jul 31 15:02:47.346870 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:47.346833    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s"]
Jul 31 15:02:47.349985 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:02:47.349955    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0424baf_32d5_4fea_820d_9f3043bf7ed3.slice/crio-c1269251db97ce41ef53f27c6b0373a99928f2045977d036b220dcaee32a5d68 WatchSource:0}: Error finding container c1269251db97ce41ef53f27c6b0373a99928f2045977d036b220dcaee32a5d68: Status 404 returned error can't find the container with id c1269251db97ce41ef53f27c6b0373a99928f2045977d036b220dcaee32a5d68
Jul 31 15:02:47.435198 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:47.435151    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s" event={"ID":"d0424baf-32d5-4fea-820d-9f3043bf7ed3","Type":"ContainerStarted","Data":"3271333fdce28fc55bf996ef26864e0d49b2b89cb82fe260072b3e5263b3881c"}
Jul 31 15:02:47.435409 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:47.435244    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s" event={"ID":"d0424baf-32d5-4fea-820d-9f3043bf7ed3","Type":"ContainerStarted","Data":"c1269251db97ce41ef53f27c6b0373a99928f2045977d036b220dcaee32a5d68"}
Jul 31 15:02:49.008776 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:49.008713    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 31 15:02:49.013032 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:49.013009    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:5000: connect: connection refused"
Jul 31 15:02:49.013500 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:49.013474    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:02:51.176819 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:51.176775    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"]
Jul 31 15:02:51.180319 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:51.180301    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:02:51.182777 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:51.182722    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 31 15:02:51.182881 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:51.182781    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 31 15:02:51.191847 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:51.191825    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"]
Jul 31 15:02:51.257814 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:51.257782    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-kserve-provision-location\") pod \"isvc-logger-predictor-754d978c8d-6gzmv\" (UID: \"4fc53b8a-3f86-47cf-af9c-96ab0b52b873\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:02:51.258012 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:51.257843    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxvls\" (UniqueName: \"kubernetes.io/projected/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-kube-api-access-jxvls\") pod \"isvc-logger-predictor-754d978c8d-6gzmv\" (UID: \"4fc53b8a-3f86-47cf-af9c-96ab0b52b873\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:02:51.258096 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:51.258044    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-cabundle-cert\") pod \"isvc-logger-predictor-754d978c8d-6gzmv\" (UID: \"4fc53b8a-3f86-47cf-af9c-96ab0b52b873\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:02:51.258180 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:51.258118    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-proxy-tls\") pod \"isvc-logger-predictor-754d978c8d-6gzmv\" (UID: \"4fc53b8a-3f86-47cf-af9c-96ab0b52b873\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:02:51.258180 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:51.258165    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-754d978c8d-6gzmv\" (UID: \"4fc53b8a-3f86-47cf-af9c-96ab0b52b873\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:02:51.359061 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:51.359026    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-proxy-tls\") pod \"isvc-logger-predictor-754d978c8d-6gzmv\" (UID: \"4fc53b8a-3f86-47cf-af9c-96ab0b52b873\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:02:51.359061 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:51.359069    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-754d978c8d-6gzmv\" (UID: \"4fc53b8a-3f86-47cf-af9c-96ab0b52b873\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:02:51.359275 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:51.359106    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-kserve-provision-location\") pod \"isvc-logger-predictor-754d978c8d-6gzmv\" (UID: \"4fc53b8a-3f86-47cf-af9c-96ab0b52b873\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:02:51.359275 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:51.359132    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxvls\" (UniqueName: \"kubernetes.io/projected/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-kube-api-access-jxvls\") pod \"isvc-logger-predictor-754d978c8d-6gzmv\" (UID: \"4fc53b8a-3f86-47cf-af9c-96ab0b52b873\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:02:51.359275 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:51.359157    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-cabundle-cert\") pod \"isvc-logger-predictor-754d978c8d-6gzmv\" (UID: \"4fc53b8a-3f86-47cf-af9c-96ab0b52b873\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:02:51.359275 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:02:51.359195    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Jul 31 15:02:51.359444 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:02:51.359284    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-proxy-tls podName:4fc53b8a-3f86-47cf-af9c-96ab0b52b873 nodeName:}" failed. No retries permitted until 2026-07-31 15:02:51.859261799 +0000 UTC m=+868.170819400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-proxy-tls") pod "isvc-logger-predictor-754d978c8d-6gzmv" (UID: "4fc53b8a-3f86-47cf-af9c-96ab0b52b873") : secret "isvc-logger-predictor-serving-cert" not found
Jul 31 15:02:51.359521 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:51.359502    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-kserve-provision-location\") pod \"isvc-logger-predictor-754d978c8d-6gzmv\" (UID: \"4fc53b8a-3f86-47cf-af9c-96ab0b52b873\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:02:51.359895 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:51.359874    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-cabundle-cert\") pod \"isvc-logger-predictor-754d978c8d-6gzmv\" (UID: \"4fc53b8a-3f86-47cf-af9c-96ab0b52b873\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:02:51.359989 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:51.359929    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-754d978c8d-6gzmv\" (UID: \"4fc53b8a-3f86-47cf-af9c-96ab0b52b873\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:02:51.368204 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:51.368175    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxvls\" (UniqueName: \"kubernetes.io/projected/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-kube-api-access-jxvls\") pod \"isvc-logger-predictor-754d978c8d-6gzmv\" (UID: \"4fc53b8a-3f86-47cf-af9c-96ab0b52b873\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:02:51.862682 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:51.862627    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-proxy-tls\") pod \"isvc-logger-predictor-754d978c8d-6gzmv\" (UID: \"4fc53b8a-3f86-47cf-af9c-96ab0b52b873\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:02:51.865340 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:51.865314    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-proxy-tls\") pod \"isvc-logger-predictor-754d978c8d-6gzmv\" (UID: \"4fc53b8a-3f86-47cf-af9c-96ab0b52b873\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:02:52.090662 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:52.090620    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:02:52.233094 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:52.233071    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"]
Jul 31 15:02:52.235636 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:02:52.235608    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc53b8a_3f86_47cf_af9c_96ab0b52b873.slice/crio-5b0accbb04c2cb43579d7f76896fa603670e7eb78f33255e39ebf03668c65d2a WatchSource:0}: Error finding container 5b0accbb04c2cb43579d7f76896fa603670e7eb78f33255e39ebf03668c65d2a: Status 404 returned error can't find the container with id 5b0accbb04c2cb43579d7f76896fa603670e7eb78f33255e39ebf03668c65d2a
Jul 31 15:02:52.461069 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:52.461042    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s_d0424baf-32d5-4fea-820d-9f3043bf7ed3/storage-initializer/0.log"
Jul 31 15:02:52.461240 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:52.461082    2559 generic.go:356] "Generic (PLEG): container finished" podID="d0424baf-32d5-4fea-820d-9f3043bf7ed3" containerID="3271333fdce28fc55bf996ef26864e0d49b2b89cb82fe260072b3e5263b3881c" exitCode=1
Jul 31 15:02:52.461240 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:52.461151    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s" event={"ID":"d0424baf-32d5-4fea-820d-9f3043bf7ed3","Type":"ContainerDied","Data":"3271333fdce28fc55bf996ef26864e0d49b2b89cb82fe260072b3e5263b3881c"}
Jul 31 15:02:52.463058 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:52.463036    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" event={"ID":"4fc53b8a-3f86-47cf-af9c-96ab0b52b873","Type":"ContainerStarted","Data":"82c5bdeed2ea2e2a0da040f4c9a18c86a7fad3d223f2c61a2dc6726473aec6bb"}
Jul 31 15:02:52.463147 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:52.463096    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" event={"ID":"4fc53b8a-3f86-47cf-af9c-96ab0b52b873","Type":"ContainerStarted","Data":"5b0accbb04c2cb43579d7f76896fa603670e7eb78f33255e39ebf03668c65d2a"}
Jul 31 15:02:53.468397 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:53.468365    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s_d0424baf-32d5-4fea-820d-9f3043bf7ed3/storage-initializer/0.log"
Jul 31 15:02:53.468774 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:53.468490    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s" event={"ID":"d0424baf-32d5-4fea-820d-9f3043bf7ed3","Type":"ContainerStarted","Data":"bcf90292b38cfc9af0ee7fce1443201d0c0a641662c88234e8f4574ef2cb4688"}
Jul 31 15:02:54.007967 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:54.007929    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 31 15:02:56.482634 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:56.482542    2559 generic.go:356] "Generic (PLEG): container finished" podID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerID="82c5bdeed2ea2e2a0da040f4c9a18c86a7fad3d223f2c61a2dc6726473aec6bb" exitCode=0
Jul 31 15:02:56.483115 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:56.482624    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" event={"ID":"4fc53b8a-3f86-47cf-af9c-96ab0b52b873","Type":"ContainerDied","Data":"82c5bdeed2ea2e2a0da040f4c9a18c86a7fad3d223f2c61a2dc6726473aec6bb"}
Jul 31 15:02:57.490277 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:57.490247    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" event={"ID":"4fc53b8a-3f86-47cf-af9c-96ab0b52b873","Type":"ContainerStarted","Data":"2cc9233491f69fac51af2640182c63f1572e39297d04252633c57765ee66414d"}
Jul 31 15:02:57.490587 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:57.490321    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" event={"ID":"4fc53b8a-3f86-47cf-af9c-96ab0b52b873","Type":"ContainerStarted","Data":"13fa20bfa9dff46701c01f14cbe25956200feac408ba486e4b081c44538d4b80"}
Jul 31 15:02:57.490587 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:57.490346    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" event={"ID":"4fc53b8a-3f86-47cf-af9c-96ab0b52b873","Type":"ContainerStarted","Data":"c5f12c8971762996101c631b85ad1e8dc65a4d51c676f5291a634eba8d3933b0"}
Jul 31 15:02:57.490696 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:57.490679    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:02:57.490780 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:57.490769    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:02:57.492236 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:57.492211    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 31 15:02:57.512749 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:57.512696    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podStartSLOduration=2.688614961 podStartE2EDuration="6.51268367s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.824068709s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:02:57.511275918 +0000 UTC m=+873.822833565" watchObservedRunningTime="2026-07-31 15:02:57.51268367 +0000 UTC m=+873.824241292"
Jul 31 15:02:58.496472 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:58.496444    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s_d0424baf-32d5-4fea-820d-9f3043bf7ed3/storage-initializer/1.log"
Jul 31 15:02:58.496871 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:58.496840    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s_d0424baf-32d5-4fea-820d-9f3043bf7ed3/storage-initializer/0.log"
Jul 31 15:02:58.496913 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:58.496873    2559 generic.go:356] "Generic (PLEG): container finished" podID="d0424baf-32d5-4fea-820d-9f3043bf7ed3" containerID="bcf90292b38cfc9af0ee7fce1443201d0c0a641662c88234e8f4574ef2cb4688" exitCode=1
Jul 31 15:02:58.496973 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:58.496951    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s" event={"ID":"d0424baf-32d5-4fea-820d-9f3043bf7ed3","Type":"ContainerDied","Data":"bcf90292b38cfc9af0ee7fce1443201d0c0a641662c88234e8f4574ef2cb4688"}
Jul 31 15:02:58.497026 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:58.497016    2559 scope.go:117] "RemoveContainer" containerID="3271333fdce28fc55bf996ef26864e0d49b2b89cb82fe260072b3e5263b3881c"
Jul 31 15:02:58.497530 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:58.497478    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:02:58.497716 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:58.497537    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 31 15:02:58.497716 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:58.497608    2559 scope.go:117] "RemoveContainer" containerID="3271333fdce28fc55bf996ef26864e0d49b2b89cb82fe260072b3e5263b3881c"
Jul 31 15:02:58.498973 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:58.498949    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:02:58.508253 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:02:58.508236    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s_kserve-ci-e2e-test(d0424baf-32d5-4fea-820d-9f3043bf7ed3)\"" pod="kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s" podUID="d0424baf-32d5-4fea-820d-9f3043bf7ed3"
Jul 31 15:02:58.508461 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:02:58.508438    2559 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s_kserve-ci-e2e-test_d0424baf-32d5-4fea-820d-9f3043bf7ed3_0 in pod sandbox c1269251db97ce41ef53f27c6b0373a99928f2045977d036b220dcaee32a5d68 from index: no such id: '3271333fdce28fc55bf996ef26864e0d49b2b89cb82fe260072b3e5263b3881c'" containerID="3271333fdce28fc55bf996ef26864e0d49b2b89cb82fe260072b3e5263b3881c"
Jul 31 15:02:58.508519 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:58.508470    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3271333fdce28fc55bf996ef26864e0d49b2b89cb82fe260072b3e5263b3881c"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s_kserve-ci-e2e-test_d0424baf-32d5-4fea-820d-9f3043bf7ed3_0 in pod sandbox c1269251db97ce41ef53f27c6b0373a99928f2045977d036b220dcaee32a5d68 from index: no such id: '3271333fdce28fc55bf996ef26864e0d49b2b89cb82fe260072b3e5263b3881c'"
Jul 31 15:02:59.008602 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:59.008555    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 31 15:02:59.012932 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:59.012900    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:5000: connect: connection refused"
Jul 31 15:02:59.013086 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:59.013032    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"
Jul 31 15:02:59.013311 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:59.013283    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:02:59.013418 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:59.013404    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"
Jul 31 15:02:59.505420 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:59.505391    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s_d0424baf-32d5-4fea-820d-9f3043bf7ed3/storage-initializer/1.log"
Jul 31 15:02:59.505991 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:59.505953    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 31 15:02:59.506305 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:02:59.506282    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:03:01.195703 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.195678    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"
Jul 31 15:03:01.237907 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.237878    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/757a6c4f-ad9a-420d-b233-449c16bc1784-kserve-provision-location\") pod \"757a6c4f-ad9a-420d-b233-449c16bc1784\" (UID: \"757a6c4f-ad9a-420d-b233-449c16bc1784\") "
Jul 31 15:03:01.238104 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.237943    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/757a6c4f-ad9a-420d-b233-449c16bc1784-proxy-tls\") pod \"757a6c4f-ad9a-420d-b233-449c16bc1784\" (UID: \"757a6c4f-ad9a-420d-b233-449c16bc1784\") "
Jul 31 15:03:01.238104 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.237973    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4qj4\" (UniqueName: \"kubernetes.io/projected/757a6c4f-ad9a-420d-b233-449c16bc1784-kube-api-access-j4qj4\") pod \"757a6c4f-ad9a-420d-b233-449c16bc1784\" (UID: \"757a6c4f-ad9a-420d-b233-449c16bc1784\") "
Jul 31 15:03:01.238104 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.237991    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/757a6c4f-ad9a-420d-b233-449c16bc1784-cabundle-cert\") pod \"757a6c4f-ad9a-420d-b233-449c16bc1784\" (UID: \"757a6c4f-ad9a-420d-b233-449c16bc1784\") "
Jul 31 15:03:01.238104 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.238034    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/757a6c4f-ad9a-420d-b233-449c16bc1784-kube-rbac-proxy-sar-config\") pod \"757a6c4f-ad9a-420d-b233-449c16bc1784\" (UID: \"757a6c4f-ad9a-420d-b233-449c16bc1784\") "
Jul 31 15:03:01.238345 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.238313    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/757a6c4f-ad9a-420d-b233-449c16bc1784-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "757a6c4f-ad9a-420d-b233-449c16bc1784" (UID: "757a6c4f-ad9a-420d-b233-449c16bc1784"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:03:01.238407 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.238355    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/757a6c4f-ad9a-420d-b233-449c16bc1784-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "757a6c4f-ad9a-420d-b233-449c16bc1784" (UID: "757a6c4f-ad9a-420d-b233-449c16bc1784"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:03:01.238407 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.238396    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/757a6c4f-ad9a-420d-b233-449c16bc1784-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "757a6c4f-ad9a-420d-b233-449c16bc1784" (UID: "757a6c4f-ad9a-420d-b233-449c16bc1784"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:03:01.240760 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.240720    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/757a6c4f-ad9a-420d-b233-449c16bc1784-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "757a6c4f-ad9a-420d-b233-449c16bc1784" (UID: "757a6c4f-ad9a-420d-b233-449c16bc1784"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:03:01.240880 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.240773    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/757a6c4f-ad9a-420d-b233-449c16bc1784-kube-api-access-j4qj4" (OuterVolumeSpecName: "kube-api-access-j4qj4") pod "757a6c4f-ad9a-420d-b233-449c16bc1784" (UID: "757a6c4f-ad9a-420d-b233-449c16bc1784"). InnerVolumeSpecName "kube-api-access-j4qj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:03:01.339091 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.338995    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/757a6c4f-ad9a-420d-b233-449c16bc1784-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:03:01.339091 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.339036    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/757a6c4f-ad9a-420d-b233-449c16bc1784-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:03:01.339091 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.339047    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/757a6c4f-ad9a-420d-b233-449c16bc1784-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:03:01.339091 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.339056    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4qj4\" (UniqueName: \"kubernetes.io/projected/757a6c4f-ad9a-420d-b233-449c16bc1784-kube-api-access-j4qj4\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:03:01.339091 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.339064    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/757a6c4f-ad9a-420d-b233-449c16bc1784-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:03:01.515385 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.515355    2559 generic.go:356] "Generic (PLEG): container finished" podID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerID="46bebb7b027a41f3ff2ae26debb7e454f2633a7fd28b49eeb10eb38b09f44545" exitCode=0
Jul 31 15:03:01.515540 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.515396    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" event={"ID":"757a6c4f-ad9a-420d-b233-449c16bc1784","Type":"ContainerDied","Data":"46bebb7b027a41f3ff2ae26debb7e454f2633a7fd28b49eeb10eb38b09f44545"}
Jul 31 15:03:01.515540 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.515418    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"
Jul 31 15:03:01.515540 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.515451    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt" event={"ID":"757a6c4f-ad9a-420d-b233-449c16bc1784","Type":"ContainerDied","Data":"eeb34f85d82e8d7456505eda9155be3adb93ca6f76701b3bbab04e2fc9328859"}
Jul 31 15:03:01.515540 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.515485    2559 scope.go:117] "RemoveContainer" containerID="46bebb7b027a41f3ff2ae26debb7e454f2633a7fd28b49eeb10eb38b09f44545"
Jul 31 15:03:01.523857 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.523832    2559 scope.go:117] "RemoveContainer" containerID="5b45920908bc6be71bbb7a264febddbc212b1c3192fdb25cde720324f7d4fb57"
Jul 31 15:03:01.531467 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.531449    2559 scope.go:117] "RemoveContainer" containerID="780888eb275bf6b249cb33633a3827049b356bf8a2ee983aeb2089831d18605a"
Jul 31 15:03:01.537192 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.537166    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"]
Jul 31 15:03:01.539101 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.539073    2559 scope.go:117] "RemoveContainer" containerID="de5b6a5a6435fe464c03af28c8ff42c659f647f361486497c403745f7b3c1ce0"
Jul 31 15:03:01.545502 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.545480    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-74bdc4498b-pdfrt"]
Jul 31 15:03:01.547013 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.546999    2559 scope.go:117] "RemoveContainer" containerID="46bebb7b027a41f3ff2ae26debb7e454f2633a7fd28b49eeb10eb38b09f44545"
Jul 31 15:03:01.547279 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:03:01.547262    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46bebb7b027a41f3ff2ae26debb7e454f2633a7fd28b49eeb10eb38b09f44545\": container with ID starting with 46bebb7b027a41f3ff2ae26debb7e454f2633a7fd28b49eeb10eb38b09f44545 not found: ID does not exist" containerID="46bebb7b027a41f3ff2ae26debb7e454f2633a7fd28b49eeb10eb38b09f44545"
Jul 31 15:03:01.547322 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.547300    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46bebb7b027a41f3ff2ae26debb7e454f2633a7fd28b49eeb10eb38b09f44545"} err="failed to get container status \"46bebb7b027a41f3ff2ae26debb7e454f2633a7fd28b49eeb10eb38b09f44545\": rpc error: code = NotFound desc = could not find container \"46bebb7b027a41f3ff2ae26debb7e454f2633a7fd28b49eeb10eb38b09f44545\": container with ID starting with 46bebb7b027a41f3ff2ae26debb7e454f2633a7fd28b49eeb10eb38b09f44545 not found: ID does not exist"
Jul 31 15:03:01.547322 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.547317    2559 scope.go:117] "RemoveContainer" containerID="5b45920908bc6be71bbb7a264febddbc212b1c3192fdb25cde720324f7d4fb57"
Jul 31 15:03:01.547554 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:03:01.547539    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b45920908bc6be71bbb7a264febddbc212b1c3192fdb25cde720324f7d4fb57\": container with ID starting with 5b45920908bc6be71bbb7a264febddbc212b1c3192fdb25cde720324f7d4fb57 not found: ID does not exist" containerID="5b45920908bc6be71bbb7a264febddbc212b1c3192fdb25cde720324f7d4fb57"
Jul 31 15:03:01.547598 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.547559    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b45920908bc6be71bbb7a264febddbc212b1c3192fdb25cde720324f7d4fb57"} err="failed to get container status \"5b45920908bc6be71bbb7a264febddbc212b1c3192fdb25cde720324f7d4fb57\": rpc error: code = NotFound desc = could not find container \"5b45920908bc6be71bbb7a264febddbc212b1c3192fdb25cde720324f7d4fb57\": container with ID starting with 5b45920908bc6be71bbb7a264febddbc212b1c3192fdb25cde720324f7d4fb57 not found: ID does not exist"
Jul 31 15:03:01.547598 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.547573    2559 scope.go:117] "RemoveContainer" containerID="780888eb275bf6b249cb33633a3827049b356bf8a2ee983aeb2089831d18605a"
Jul 31 15:03:01.547823 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:03:01.547803    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"780888eb275bf6b249cb33633a3827049b356bf8a2ee983aeb2089831d18605a\": container with ID starting with 780888eb275bf6b249cb33633a3827049b356bf8a2ee983aeb2089831d18605a not found: ID does not exist" containerID="780888eb275bf6b249cb33633a3827049b356bf8a2ee983aeb2089831d18605a"
Jul 31 15:03:01.547896 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.547832    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"780888eb275bf6b249cb33633a3827049b356bf8a2ee983aeb2089831d18605a"} err="failed to get container status \"780888eb275bf6b249cb33633a3827049b356bf8a2ee983aeb2089831d18605a\": rpc error: code = NotFound desc = could not find container \"780888eb275bf6b249cb33633a3827049b356bf8a2ee983aeb2089831d18605a\": container with ID starting with 780888eb275bf6b249cb33633a3827049b356bf8a2ee983aeb2089831d18605a not found: ID does not exist"
Jul 31 15:03:01.547896 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.547854    2559 scope.go:117] "RemoveContainer" containerID="de5b6a5a6435fe464c03af28c8ff42c659f647f361486497c403745f7b3c1ce0"
Jul 31 15:03:01.548092 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:03:01.548076    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de5b6a5a6435fe464c03af28c8ff42c659f647f361486497c403745f7b3c1ce0\": container with ID starting with de5b6a5a6435fe464c03af28c8ff42c659f647f361486497c403745f7b3c1ce0 not found: ID does not exist" containerID="de5b6a5a6435fe464c03af28c8ff42c659f647f361486497c403745f7b3c1ce0"
Jul 31 15:03:01.548142 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:01.548098    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5b6a5a6435fe464c03af28c8ff42c659f647f361486497c403745f7b3c1ce0"} err="failed to get container status \"de5b6a5a6435fe464c03af28c8ff42c659f647f361486497c403745f7b3c1ce0\": rpc error: code = NotFound desc = could not find container \"de5b6a5a6435fe464c03af28c8ff42c659f647f361486497c403745f7b3c1ce0\": container with ID starting with de5b6a5a6435fe464c03af28c8ff42c659f647f361486497c403745f7b3c1ce0 not found: ID does not exist"
Jul 31 15:03:02.220671 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:02.220640    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" path="/var/lib/kubelet/pods/757a6c4f-ad9a-420d-b233-449c16bc1784/volumes"
Jul 31 15:03:04.356367 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.356335    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s"]
Jul 31 15:03:04.394899 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.394864    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"]
Jul 31 15:03:04.395220 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.395192    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx" podUID="c79515a4-af1a-4622-be1c-8ed4d13e3302" containerName="kserve-container" containerID="cri-o://a68c6e57662858e0068d7ed522bfbf91f7152255938907708384aa66e6075480" gracePeriod=30
Jul 31 15:03:04.395326 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.395196    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx" podUID="c79515a4-af1a-4622-be1c-8ed4d13e3302" containerName="kube-rbac-proxy" containerID="cri-o://b4d29867c8d702f52f75468bf4801b7869bed6a2593399ad44697e69afd00c50" gracePeriod=30
Jul 31 15:03:04.510598 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.510575    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:03:04.511216 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.511189    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 31 15:03:04.511569 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.511537    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:03:04.515569 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.515552    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s_d0424baf-32d5-4fea-820d-9f3043bf7ed3/storage-initializer/1.log"
Jul 31 15:03:04.517669 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.517652    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s"
Jul 31 15:03:04.529594 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.529578    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s_d0424baf-32d5-4fea-820d-9f3043bf7ed3/storage-initializer/1.log"
Jul 31 15:03:04.529689 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.529670    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s"
Jul 31 15:03:04.529752 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.529684    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s" event={"ID":"d0424baf-32d5-4fea-820d-9f3043bf7ed3","Type":"ContainerDied","Data":"c1269251db97ce41ef53f27c6b0373a99928f2045977d036b220dcaee32a5d68"}
Jul 31 15:03:04.529798 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.529778    2559 scope.go:117] "RemoveContainer" containerID="bcf90292b38cfc9af0ee7fce1443201d0c0a641662c88234e8f4574ef2cb4688"
Jul 31 15:03:04.531950 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.531933    2559 generic.go:356] "Generic (PLEG): container finished" podID="c79515a4-af1a-4622-be1c-8ed4d13e3302" containerID="b4d29867c8d702f52f75468bf4801b7869bed6a2593399ad44697e69afd00c50" exitCode=2
Jul 31 15:03:04.532036 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.531964    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx" event={"ID":"c79515a4-af1a-4622-be1c-8ed4d13e3302","Type":"ContainerDied","Data":"b4d29867c8d702f52f75468bf4801b7869bed6a2593399ad44697e69afd00c50"}
Jul 31 15:03:04.565116 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.565070    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tz89g\" (UniqueName: \"kubernetes.io/projected/d0424baf-32d5-4fea-820d-9f3043bf7ed3-kube-api-access-tz89g\") pod \"d0424baf-32d5-4fea-820d-9f3043bf7ed3\" (UID: \"d0424baf-32d5-4fea-820d-9f3043bf7ed3\") "
Jul 31 15:03:04.565116 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.565123    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0424baf-32d5-4fea-820d-9f3043bf7ed3-kube-rbac-proxy-sar-config\") pod \"d0424baf-32d5-4fea-820d-9f3043bf7ed3\" (UID: \"d0424baf-32d5-4fea-820d-9f3043bf7ed3\") "
Jul 31 15:03:04.565335 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.565186    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0424baf-32d5-4fea-820d-9f3043bf7ed3-kserve-provision-location\") pod \"d0424baf-32d5-4fea-820d-9f3043bf7ed3\" (UID: \"d0424baf-32d5-4fea-820d-9f3043bf7ed3\") "
Jul 31 15:03:04.565335 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.565226    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0424baf-32d5-4fea-820d-9f3043bf7ed3-cabundle-cert\") pod \"d0424baf-32d5-4fea-820d-9f3043bf7ed3\" (UID: \"d0424baf-32d5-4fea-820d-9f3043bf7ed3\") "
Jul 31 15:03:04.565335 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.565254    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0424baf-32d5-4fea-820d-9f3043bf7ed3-proxy-tls\") pod \"d0424baf-32d5-4fea-820d-9f3043bf7ed3\" (UID: \"d0424baf-32d5-4fea-820d-9f3043bf7ed3\") "
Jul 31 15:03:04.565550 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.565431    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0424baf-32d5-4fea-820d-9f3043bf7ed3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d0424baf-32d5-4fea-820d-9f3043bf7ed3" (UID: "d0424baf-32d5-4fea-820d-9f3043bf7ed3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:03:04.565607 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.565569    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0424baf-32d5-4fea-820d-9f3043bf7ed3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d0424baf-32d5-4fea-820d-9f3043bf7ed3" (UID: "d0424baf-32d5-4fea-820d-9f3043bf7ed3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:03:04.565684 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.565661    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0424baf-32d5-4fea-820d-9f3043bf7ed3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d0424baf-32d5-4fea-820d-9f3043bf7ed3" (UID: "d0424baf-32d5-4fea-820d-9f3043bf7ed3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:03:04.565800 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.565772    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0424baf-32d5-4fea-820d-9f3043bf7ed3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:03:04.565800 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.565795    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0424baf-32d5-4fea-820d-9f3043bf7ed3-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:03:04.565995 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.565810    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0424baf-32d5-4fea-820d-9f3043bf7ed3-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:03:04.567368 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.567343    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0424baf-32d5-4fea-820d-9f3043bf7ed3-kube-api-access-tz89g" (OuterVolumeSpecName: "kube-api-access-tz89g") pod "d0424baf-32d5-4fea-820d-9f3043bf7ed3" (UID: "d0424baf-32d5-4fea-820d-9f3043bf7ed3"). InnerVolumeSpecName "kube-api-access-tz89g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:03:04.567556 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.567537    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0424baf-32d5-4fea-820d-9f3043bf7ed3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d0424baf-32d5-4fea-820d-9f3043bf7ed3" (UID: "d0424baf-32d5-4fea-820d-9f3043bf7ed3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:03:04.666427 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.666341    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0424baf-32d5-4fea-820d-9f3043bf7ed3-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:03:04.666427 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.666373    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz89g\" (UniqueName: \"kubernetes.io/projected/d0424baf-32d5-4fea-820d-9f3043bf7ed3-kube-api-access-tz89g\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:03:04.868600 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.868564    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s"]
Jul 31 15:03:04.873009 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:04.872978    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-10a884-predictor-86cc5c8f8f-8sb7s"]
Jul 31 15:03:05.036042 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:05.035949    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx" podUID="c79515a4-af1a-4622-be1c-8ed4d13e3302" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 31 15:03:06.221398 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:06.221364    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0424baf-32d5-4fea-820d-9f3043bf7ed3" path="/var/lib/kubelet/pods/d0424baf-32d5-4fea-820d-9f3043bf7ed3/volumes"
Jul 31 15:03:09.144869 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.144846    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"
Jul 31 15:03:09.205666 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.205639    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c79515a4-af1a-4622-be1c-8ed4d13e3302-kube-rbac-proxy-sar-config\") pod \"c79515a4-af1a-4622-be1c-8ed4d13e3302\" (UID: \"c79515a4-af1a-4622-be1c-8ed4d13e3302\") "
Jul 31 15:03:09.205912 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.205679    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztgdg\" (UniqueName: \"kubernetes.io/projected/c79515a4-af1a-4622-be1c-8ed4d13e3302-kube-api-access-ztgdg\") pod \"c79515a4-af1a-4622-be1c-8ed4d13e3302\" (UID: \"c79515a4-af1a-4622-be1c-8ed4d13e3302\") "
Jul 31 15:03:09.205912 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.205701    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c79515a4-af1a-4622-be1c-8ed4d13e3302-proxy-tls\") pod \"c79515a4-af1a-4622-be1c-8ed4d13e3302\" (UID: \"c79515a4-af1a-4622-be1c-8ed4d13e3302\") "
Jul 31 15:03:09.205912 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.205753    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c79515a4-af1a-4622-be1c-8ed4d13e3302-cabundle-cert\") pod \"c79515a4-af1a-4622-be1c-8ed4d13e3302\" (UID: \"c79515a4-af1a-4622-be1c-8ed4d13e3302\") "
Jul 31 15:03:09.205912 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.205797    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c79515a4-af1a-4622-be1c-8ed4d13e3302-kserve-provision-location\") pod \"c79515a4-af1a-4622-be1c-8ed4d13e3302\" (UID: \"c79515a4-af1a-4622-be1c-8ed4d13e3302\") "
Jul 31 15:03:09.206139 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.206089    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c79515a4-af1a-4622-be1c-8ed4d13e3302-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c79515a4-af1a-4622-be1c-8ed4d13e3302" (UID: "c79515a4-af1a-4622-be1c-8ed4d13e3302"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:03:09.206216 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.206190    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c79515a4-af1a-4622-be1c-8ed4d13e3302-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c79515a4-af1a-4622-be1c-8ed4d13e3302" (UID: "c79515a4-af1a-4622-be1c-8ed4d13e3302"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:03:09.206268 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.206218    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c79515a4-af1a-4622-be1c-8ed4d13e3302-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c79515a4-af1a-4622-be1c-8ed4d13e3302" (UID: "c79515a4-af1a-4622-be1c-8ed4d13e3302"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:03:09.207937 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.207914    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c79515a4-af1a-4622-be1c-8ed4d13e3302-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c79515a4-af1a-4622-be1c-8ed4d13e3302" (UID: "c79515a4-af1a-4622-be1c-8ed4d13e3302"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:03:09.208041 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.207978    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c79515a4-af1a-4622-be1c-8ed4d13e3302-kube-api-access-ztgdg" (OuterVolumeSpecName: "kube-api-access-ztgdg") pod "c79515a4-af1a-4622-be1c-8ed4d13e3302" (UID: "c79515a4-af1a-4622-be1c-8ed4d13e3302"). InnerVolumeSpecName "kube-api-access-ztgdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:03:09.307460 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.307369    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztgdg\" (UniqueName: \"kubernetes.io/projected/c79515a4-af1a-4622-be1c-8ed4d13e3302-kube-api-access-ztgdg\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:03:09.307630 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.307528    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c79515a4-af1a-4622-be1c-8ed4d13e3302-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:03:09.307630 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.307556    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c79515a4-af1a-4622-be1c-8ed4d13e3302-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:03:09.307630 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.307576    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c79515a4-af1a-4622-be1c-8ed4d13e3302-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:03:09.307630 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.307591    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c79515a4-af1a-4622-be1c-8ed4d13e3302-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:03:09.553283 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.553250    2559 generic.go:356] "Generic (PLEG): container finished" podID="c79515a4-af1a-4622-be1c-8ed4d13e3302" containerID="a68c6e57662858e0068d7ed522bfbf91f7152255938907708384aa66e6075480" exitCode=0
Jul 31 15:03:09.553446 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.553311    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"
Jul 31 15:03:09.553446 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.553327    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx" event={"ID":"c79515a4-af1a-4622-be1c-8ed4d13e3302","Type":"ContainerDied","Data":"a68c6e57662858e0068d7ed522bfbf91f7152255938907708384aa66e6075480"}
Jul 31 15:03:09.553446 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.553389    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx" event={"ID":"c79515a4-af1a-4622-be1c-8ed4d13e3302","Type":"ContainerDied","Data":"b1c67e018409e1cc5e76f53c55b14d55e49b8dc075ed26140898920dc01d08a9"}
Jul 31 15:03:09.553446 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.553418    2559 scope.go:117] "RemoveContainer" containerID="b4d29867c8d702f52f75468bf4801b7869bed6a2593399ad44697e69afd00c50"
Jul 31 15:03:09.562002 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.561984    2559 scope.go:117] "RemoveContainer" containerID="a68c6e57662858e0068d7ed522bfbf91f7152255938907708384aa66e6075480"
Jul 31 15:03:09.569955 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.569937    2559 scope.go:117] "RemoveContainer" containerID="ee7b17723fa5a6766cb331ebfe932dc2edf75be7f3eb7bc92d4ccddb8c3ab06e"
Jul 31 15:03:09.576001 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.575979    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"]
Jul 31 15:03:09.578094 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.578072    2559 scope.go:117] "RemoveContainer" containerID="b4d29867c8d702f52f75468bf4801b7869bed6a2593399ad44697e69afd00c50"
Jul 31 15:03:09.578412 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:03:09.578387    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4d29867c8d702f52f75468bf4801b7869bed6a2593399ad44697e69afd00c50\": container with ID starting with b4d29867c8d702f52f75468bf4801b7869bed6a2593399ad44697e69afd00c50 not found: ID does not exist" containerID="b4d29867c8d702f52f75468bf4801b7869bed6a2593399ad44697e69afd00c50"
Jul 31 15:03:09.578487 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.578421    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4d29867c8d702f52f75468bf4801b7869bed6a2593399ad44697e69afd00c50"} err="failed to get container status \"b4d29867c8d702f52f75468bf4801b7869bed6a2593399ad44697e69afd00c50\": rpc error: code = NotFound desc = could not find container \"b4d29867c8d702f52f75468bf4801b7869bed6a2593399ad44697e69afd00c50\": container with ID starting with b4d29867c8d702f52f75468bf4801b7869bed6a2593399ad44697e69afd00c50 not found: ID does not exist"
Jul 31 15:03:09.578487 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.578447    2559 scope.go:117] "RemoveContainer" containerID="a68c6e57662858e0068d7ed522bfbf91f7152255938907708384aa66e6075480"
Jul 31 15:03:09.578687 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:03:09.578669    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a68c6e57662858e0068d7ed522bfbf91f7152255938907708384aa66e6075480\": container with ID starting with a68c6e57662858e0068d7ed522bfbf91f7152255938907708384aa66e6075480 not found: ID does not exist" containerID="a68c6e57662858e0068d7ed522bfbf91f7152255938907708384aa66e6075480"
Jul 31 15:03:09.578751 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.578697    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a68c6e57662858e0068d7ed522bfbf91f7152255938907708384aa66e6075480"} err="failed to get container status \"a68c6e57662858e0068d7ed522bfbf91f7152255938907708384aa66e6075480\": rpc error: code = NotFound desc = could not find container \"a68c6e57662858e0068d7ed522bfbf91f7152255938907708384aa66e6075480\": container with ID starting with a68c6e57662858e0068d7ed522bfbf91f7152255938907708384aa66e6075480 not found: ID does not exist"
Jul 31 15:03:09.578751 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.578715    2559 scope.go:117] "RemoveContainer" containerID="ee7b17723fa5a6766cb331ebfe932dc2edf75be7f3eb7bc92d4ccddb8c3ab06e"
Jul 31 15:03:09.578960 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:03:09.578943    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee7b17723fa5a6766cb331ebfe932dc2edf75be7f3eb7bc92d4ccddb8c3ab06e\": container with ID starting with ee7b17723fa5a6766cb331ebfe932dc2edf75be7f3eb7bc92d4ccddb8c3ab06e not found: ID does not exist" containerID="ee7b17723fa5a6766cb331ebfe932dc2edf75be7f3eb7bc92d4ccddb8c3ab06e"
Jul 31 15:03:09.579026 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.578968    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee7b17723fa5a6766cb331ebfe932dc2edf75be7f3eb7bc92d4ccddb8c3ab06e"} err="failed to get container status \"ee7b17723fa5a6766cb331ebfe932dc2edf75be7f3eb7bc92d4ccddb8c3ab06e\": rpc error: code = NotFound desc = could not find container \"ee7b17723fa5a6766cb331ebfe932dc2edf75be7f3eb7bc92d4ccddb8c3ab06e\": container with ID starting with ee7b17723fa5a6766cb331ebfe932dc2edf75be7f3eb7bc92d4ccddb8c3ab06e not found: ID does not exist"
Jul 31 15:03:09.582047 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.582026    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-10a884-predictor-565495c8c8-qvvlx"]
Jul 31 15:03:09.587177 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.587155    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl"]
Jul 31 15:03:09.587437 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.587426    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c79515a4-af1a-4622-be1c-8ed4d13e3302" containerName="storage-initializer"
Jul 31 15:03:09.587490 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.587439    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c79515a4-af1a-4622-be1c-8ed4d13e3302" containerName="storage-initializer"
Jul 31 15:03:09.587490 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.587449    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c79515a4-af1a-4622-be1c-8ed4d13e3302" containerName="kube-rbac-proxy"
Jul 31 15:03:09.587490 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.587455    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c79515a4-af1a-4622-be1c-8ed4d13e3302" containerName="kube-rbac-proxy"
Jul 31 15:03:09.587490 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.587463    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c79515a4-af1a-4622-be1c-8ed4d13e3302" containerName="kserve-container"
Jul 31 15:03:09.587490 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.587468    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c79515a4-af1a-4622-be1c-8ed4d13e3302" containerName="kserve-container"
Jul 31 15:03:09.587490 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.587474    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0424baf-32d5-4fea-820d-9f3043bf7ed3" containerName="storage-initializer"
Jul 31 15:03:09.587490 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.587480    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0424baf-32d5-4fea-820d-9f3043bf7ed3" containerName="storage-initializer"
Jul 31 15:03:09.587490 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.587486    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="kserve-container"
Jul 31 15:03:09.587490 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.587491    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="kserve-container"
Jul 31 15:03:09.587797 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.587498    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="agent"
Jul 31 15:03:09.587797 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.587503    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="agent"
Jul 31 15:03:09.587797 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.587512    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="kube-rbac-proxy"
Jul 31 15:03:09.587797 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.587517    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="kube-rbac-proxy"
Jul 31 15:03:09.587797 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.587527    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="storage-initializer"
Jul 31 15:03:09.587797 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.587533    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="storage-initializer"
Jul 31 15:03:09.587797 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.587579    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c79515a4-af1a-4622-be1c-8ed4d13e3302" containerName="kube-rbac-proxy"
Jul 31 15:03:09.587797 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.587587    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0424baf-32d5-4fea-820d-9f3043bf7ed3" containerName="storage-initializer"
Jul 31 15:03:09.587797 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.587592    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="agent"
Jul 31 15:03:09.587797 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.587599    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="kube-rbac-proxy"
Jul 31 15:03:09.587797 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.587605    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="757a6c4f-ad9a-420d-b233-449c16bc1784" containerName="kserve-container"
Jul 31 15:03:09.587797 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.587612    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c79515a4-af1a-4622-be1c-8ed4d13e3302" containerName="kserve-container"
Jul 31 15:03:09.587797 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.587663    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0424baf-32d5-4fea-820d-9f3043bf7ed3" containerName="storage-initializer"
Jul 31 15:03:09.587797 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.587668    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0424baf-32d5-4fea-820d-9f3043bf7ed3" containerName="storage-initializer"
Jul 31 15:03:09.587797 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.587708    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0424baf-32d5-4fea-820d-9f3043bf7ed3" containerName="storage-initializer"
Jul 31 15:03:09.592474 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.592457    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl"
Jul 31 15:03:09.595396 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.595378    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-761a61\""
Jul 31 15:03:09.596128 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.596113    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-761a61-dockercfg-mg2rs\""
Jul 31 15:03:09.596210 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.596180    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-761a61-predictor-serving-cert\""
Jul 31 15:03:09.596276 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.596244    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-761a61-kube-rbac-proxy-sar-config\""
Jul 31 15:03:09.601419 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.601399    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl"]
Jul 31 15:03:09.710685 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.710649    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2e0c576-e7d0-45d7-842d-a890f0cec516-cabundle-cert\") pod \"isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl\" (UID: \"e2e0c576-e7d0-45d7-842d-a890f0cec516\") " pod="kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl"
Jul 31 15:03:09.710685 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.710690    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zrdr\" (UniqueName: \"kubernetes.io/projected/e2e0c576-e7d0-45d7-842d-a890f0cec516-kube-api-access-8zrdr\") pod \"isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl\" (UID: \"e2e0c576-e7d0-45d7-842d-a890f0cec516\") " pod="kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl"
Jul 31 15:03:09.710910 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.710718    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2e0c576-e7d0-45d7-842d-a890f0cec516-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl\" (UID: \"e2e0c576-e7d0-45d7-842d-a890f0cec516\") " pod="kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl"
Jul 31 15:03:09.710910 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.710770    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2e0c576-e7d0-45d7-842d-a890f0cec516-proxy-tls\") pod \"isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl\" (UID: \"e2e0c576-e7d0-45d7-842d-a890f0cec516\") " pod="kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl"
Jul 31 15:03:09.710910 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.710805    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2e0c576-e7d0-45d7-842d-a890f0cec516-kserve-provision-location\") pod \"isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl\" (UID: \"e2e0c576-e7d0-45d7-842d-a890f0cec516\") " pod="kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl"
Jul 31 15:03:09.811625 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.811528    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2e0c576-e7d0-45d7-842d-a890f0cec516-cabundle-cert\") pod \"isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl\" (UID: \"e2e0c576-e7d0-45d7-842d-a890f0cec516\") " pod="kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl"
Jul 31 15:03:09.811625 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.811573    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zrdr\" (UniqueName: \"kubernetes.io/projected/e2e0c576-e7d0-45d7-842d-a890f0cec516-kube-api-access-8zrdr\") pod \"isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl\" (UID: \"e2e0c576-e7d0-45d7-842d-a890f0cec516\") " pod="kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl"
Jul 31 15:03:09.811887 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.811782    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2e0c576-e7d0-45d7-842d-a890f0cec516-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl\" (UID: \"e2e0c576-e7d0-45d7-842d-a890f0cec516\") " pod="kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl"
Jul 31 15:03:09.811887 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.811829    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2e0c576-e7d0-45d7-842d-a890f0cec516-proxy-tls\") pod \"isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl\" (UID: \"e2e0c576-e7d0-45d7-842d-a890f0cec516\") " pod="kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl"
Jul 31 15:03:09.811887 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.811881    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2e0c576-e7d0-45d7-842d-a890f0cec516-kserve-provision-location\") pod \"isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl\" (UID: \"e2e0c576-e7d0-45d7-842d-a890f0cec516\") " pod="kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl"
Jul 31 15:03:09.812235 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.812214    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2e0c576-e7d0-45d7-842d-a890f0cec516-kserve-provision-location\") pod \"isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl\" (UID: \"e2e0c576-e7d0-45d7-842d-a890f0cec516\") " pod="kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl"
Jul 31 15:03:09.812324 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.812306    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2e0c576-e7d0-45d7-842d-a890f0cec516-cabundle-cert\") pod \"isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl\" (UID: \"e2e0c576-e7d0-45d7-842d-a890f0cec516\") " pod="kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl"
Jul 31 15:03:09.812448 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.812431    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2e0c576-e7d0-45d7-842d-a890f0cec516-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl\" (UID: \"e2e0c576-e7d0-45d7-842d-a890f0cec516\") " pod="kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl"
Jul 31 15:03:09.814390 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.814374    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2e0c576-e7d0-45d7-842d-a890f0cec516-proxy-tls\") pod \"isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl\" (UID: \"e2e0c576-e7d0-45d7-842d-a890f0cec516\") " pod="kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl"
Jul 31 15:03:09.820637 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.820608    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zrdr\" (UniqueName: \"kubernetes.io/projected/e2e0c576-e7d0-45d7-842d-a890f0cec516-kube-api-access-8zrdr\") pod \"isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl\" (UID: \"e2e0c576-e7d0-45d7-842d-a890f0cec516\") " pod="kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl"
Jul 31 15:03:09.902707 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:09.902671    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl"
Jul 31 15:03:10.039249 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:10.039227    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl"]
Jul 31 15:03:10.041022 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:03:10.040996    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e0c576_e7d0_45d7_842d_a890f0cec516.slice/crio-196783deaaaecc679826386625bb54bafd6963ffd7c4d5eb1a9645cdfedae494 WatchSource:0}: Error finding container 196783deaaaecc679826386625bb54bafd6963ffd7c4d5eb1a9645cdfedae494: Status 404 returned error can't find the container with id 196783deaaaecc679826386625bb54bafd6963ffd7c4d5eb1a9645cdfedae494
Jul 31 15:03:10.225565 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:10.225532    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c79515a4-af1a-4622-be1c-8ed4d13e3302" path="/var/lib/kubelet/pods/c79515a4-af1a-4622-be1c-8ed4d13e3302/volumes"
Jul 31 15:03:10.559477 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:10.559440    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl" event={"ID":"e2e0c576-e7d0-45d7-842d-a890f0cec516","Type":"ContainerStarted","Data":"7db10fe334e7e4d5e67122b2ee2f2f320987b1d1eb3013d78cce5eb8ef750752"}
Jul 31 15:03:10.559678 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:10.559516    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl" event={"ID":"e2e0c576-e7d0-45d7-842d-a890f0cec516","Type":"ContainerStarted","Data":"196783deaaaecc679826386625bb54bafd6963ffd7c4d5eb1a9645cdfedae494"}
Jul 31 15:03:14.512000 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:14.511958    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 31 15:03:14.512444 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:14.512414    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:03:15.585259 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:15.585184    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl_e2e0c576-e7d0-45d7-842d-a890f0cec516/storage-initializer/0.log"
Jul 31 15:03:15.585259 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:15.585225    2559 generic.go:356] "Generic (PLEG): container finished" podID="e2e0c576-e7d0-45d7-842d-a890f0cec516" containerID="7db10fe334e7e4d5e67122b2ee2f2f320987b1d1eb3013d78cce5eb8ef750752" exitCode=1
Jul 31 15:03:15.585649 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:15.585271    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl" event={"ID":"e2e0c576-e7d0-45d7-842d-a890f0cec516","Type":"ContainerDied","Data":"7db10fe334e7e4d5e67122b2ee2f2f320987b1d1eb3013d78cce5eb8ef750752"}
Jul 31 15:03:16.590769 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:16.590715    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl_e2e0c576-e7d0-45d7-842d-a890f0cec516/storage-initializer/0.log"
Jul 31 15:03:16.591153 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:16.590837    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl" event={"ID":"e2e0c576-e7d0-45d7-842d-a890f0cec516","Type":"ContainerStarted","Data":"60437387821fbce48250bf6d199194d5848e032787478c813936ebecdca05ef4"}
Jul 31 15:03:19.594725 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:19.594688    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl"]
Jul 31 15:03:19.595219 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:19.594974    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl" podUID="e2e0c576-e7d0-45d7-842d-a890f0cec516" containerName="storage-initializer" containerID="cri-o://60437387821fbce48250bf6d199194d5848e032787478c813936ebecdca05ef4" gracePeriod=30
Jul 31 15:03:19.604484 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:19.604459    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl_e2e0c576-e7d0-45d7-842d-a890f0cec516/storage-initializer/1.log"
Jul 31 15:03:19.604969 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:19.604952    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl_e2e0c576-e7d0-45d7-842d-a890f0cec516/storage-initializer/0.log"
Jul 31 15:03:19.605036 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:19.604987    2559 generic.go:356] "Generic (PLEG): container finished" podID="e2e0c576-e7d0-45d7-842d-a890f0cec516" containerID="60437387821fbce48250bf6d199194d5848e032787478c813936ebecdca05ef4" exitCode=1
Jul 31 15:03:19.605073 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:19.605032    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl" event={"ID":"e2e0c576-e7d0-45d7-842d-a890f0cec516","Type":"ContainerDied","Data":"60437387821fbce48250bf6d199194d5848e032787478c813936ebecdca05ef4"}
Jul 31 15:03:19.605112 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:19.605082    2559 scope.go:117] "RemoveContainer" containerID="7db10fe334e7e4d5e67122b2ee2f2f320987b1d1eb3013d78cce5eb8ef750752"
Jul 31 15:03:19.725476 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:19.725455    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl_e2e0c576-e7d0-45d7-842d-a890f0cec516/storage-initializer/1.log"
Jul 31 15:03:19.728807 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:19.728788    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl"
Jul 31 15:03:19.891718 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:19.891624    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2e0c576-e7d0-45d7-842d-a890f0cec516-cabundle-cert\") pod \"e2e0c576-e7d0-45d7-842d-a890f0cec516\" (UID: \"e2e0c576-e7d0-45d7-842d-a890f0cec516\") "
Jul 31 15:03:19.891718 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:19.891662    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2e0c576-e7d0-45d7-842d-a890f0cec516-kube-rbac-proxy-sar-config\") pod \"e2e0c576-e7d0-45d7-842d-a890f0cec516\" (UID: \"e2e0c576-e7d0-45d7-842d-a890f0cec516\") "
Jul 31 15:03:19.891718 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:19.891692    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2e0c576-e7d0-45d7-842d-a890f0cec516-kserve-provision-location\") pod \"e2e0c576-e7d0-45d7-842d-a890f0cec516\" (UID: \"e2e0c576-e7d0-45d7-842d-a890f0cec516\") "
Jul 31 15:03:19.892035 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:19.891798    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zrdr\" (UniqueName: \"kubernetes.io/projected/e2e0c576-e7d0-45d7-842d-a890f0cec516-kube-api-access-8zrdr\") pod \"e2e0c576-e7d0-45d7-842d-a890f0cec516\" (UID: \"e2e0c576-e7d0-45d7-842d-a890f0cec516\") "
Jul 31 15:03:19.892035 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:19.891833    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2e0c576-e7d0-45d7-842d-a890f0cec516-proxy-tls\") pod \"e2e0c576-e7d0-45d7-842d-a890f0cec516\" (UID: \"e2e0c576-e7d0-45d7-842d-a890f0cec516\") "
Jul 31 15:03:19.892116 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:19.892026    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2e0c576-e7d0-45d7-842d-a890f0cec516-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e2e0c576-e7d0-45d7-842d-a890f0cec516" (UID: "e2e0c576-e7d0-45d7-842d-a890f0cec516"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:03:19.892116 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:19.892107    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2e0c576-e7d0-45d7-842d-a890f0cec516-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e2e0c576-e7d0-45d7-842d-a890f0cec516" (UID: "e2e0c576-e7d0-45d7-842d-a890f0cec516"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:03:19.892179 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:19.892119    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2e0c576-e7d0-45d7-842d-a890f0cec516-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e2e0c576-e7d0-45d7-842d-a890f0cec516" (UID: "e2e0c576-e7d0-45d7-842d-a890f0cec516"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:03:19.894060 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:19.894030    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2e0c576-e7d0-45d7-842d-a890f0cec516-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e2e0c576-e7d0-45d7-842d-a890f0cec516" (UID: "e2e0c576-e7d0-45d7-842d-a890f0cec516"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:03:19.894167 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:19.894083    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e0c576-e7d0-45d7-842d-a890f0cec516-kube-api-access-8zrdr" (OuterVolumeSpecName: "kube-api-access-8zrdr") pod "e2e0c576-e7d0-45d7-842d-a890f0cec516" (UID: "e2e0c576-e7d0-45d7-842d-a890f0cec516"). InnerVolumeSpecName "kube-api-access-8zrdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:03:19.992956 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:19.992920    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2e0c576-e7d0-45d7-842d-a890f0cec516-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:03:19.992956 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:19.992949    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2e0c576-e7d0-45d7-842d-a890f0cec516-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:03:19.992956 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:19.992958    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2e0c576-e7d0-45d7-842d-a890f0cec516-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:03:19.992956 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:19.992966    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zrdr\" (UniqueName: \"kubernetes.io/projected/e2e0c576-e7d0-45d7-842d-a890f0cec516-kube-api-access-8zrdr\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:03:19.993234 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:19.992975    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2e0c576-e7d0-45d7-842d-a890f0cec516-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:03:20.609679 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:20.609650    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl_e2e0c576-e7d0-45d7-842d-a890f0cec516/storage-initializer/1.log"
Jul 31 15:03:20.610106 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:20.609756    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl" event={"ID":"e2e0c576-e7d0-45d7-842d-a890f0cec516","Type":"ContainerDied","Data":"196783deaaaecc679826386625bb54bafd6963ffd7c4d5eb1a9645cdfedae494"}
Jul 31 15:03:20.610106 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:20.609774    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl"
Jul 31 15:03:20.610106 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:20.609821    2559 scope.go:117] "RemoveContainer" containerID="60437387821fbce48250bf6d199194d5848e032787478c813936ebecdca05ef4"
Jul 31 15:03:20.643855 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:20.643820    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl"]
Jul 31 15:03:20.648559 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:20.648532    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-761a61-predictor-7bfc57db67-6wlhl"]
Jul 31 15:03:22.220470 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:22.220439    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2e0c576-e7d0-45d7-842d-a890f0cec516" path="/var/lib/kubelet/pods/e2e0c576-e7d0-45d7-842d-a890f0cec516/volumes"
Jul 31 15:03:24.511522 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:24.511424    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 31 15:03:24.512032 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:24.512005    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:03:34.511913 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:34.511868    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 31 15:03:34.512417 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:34.512381    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:03:44.511438 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:44.511390    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 31 15:03:44.511935 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:44.511909    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:03:54.511655 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:54.511610    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 31 15:03:54.512203 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:03:54.512127    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:04:04.511815 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:04.511774    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 31 15:04:04.512335 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:04.512297    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:04:14.512349 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:14.512317    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:04:14.514828 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:14.512616    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:04:26.171083 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.171045    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-kl99q_b189726d-1b57-4ec4-876d-37850bb398d9/kserve-container/0.log"
Jul 31 15:04:26.354340 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.354310    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"]
Jul 31 15:04:26.354601 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.354565    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="kserve-container" containerID="cri-o://c5f12c8971762996101c631b85ad1e8dc65a4d51c676f5291a634eba8d3933b0" gracePeriod=30
Jul 31 15:04:26.354763 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.354620    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="kube-rbac-proxy" containerID="cri-o://13fa20bfa9dff46701c01f14cbe25956200feac408ba486e4b081c44538d4b80" gracePeriod=30
Jul 31 15:04:26.354763 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.354566    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="agent" containerID="cri-o://2cc9233491f69fac51af2640182c63f1572e39297d04252633c57765ee66414d" gracePeriod=30
Jul 31 15:04:26.415324 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.415291    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q"]
Jul 31 15:04:26.415557 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.415532    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q" podUID="b189726d-1b57-4ec4-876d-37850bb398d9" containerName="kserve-container" containerID="cri-o://38b1661ddfdb2d39e050c0ba049f1828af9aceb003de7269fbaa8614d48ccf16" gracePeriod=30
Jul 31 15:04:26.415659 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.415569    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q" podUID="b189726d-1b57-4ec4-876d-37850bb398d9" containerName="kube-rbac-proxy" containerID="cri-o://3c1c163e996b4750ecf493a4231b8e62723587341dffcda0aaed2a2c8de72faf" gracePeriod=30
Jul 31 15:04:26.454555 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.454528    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"]
Jul 31 15:04:26.454838 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.454826    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2e0c576-e7d0-45d7-842d-a890f0cec516" containerName="storage-initializer"
Jul 31 15:04:26.454880 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.454840    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2e0c576-e7d0-45d7-842d-a890f0cec516" containerName="storage-initializer"
Jul 31 15:04:26.454880 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.454853    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2e0c576-e7d0-45d7-842d-a890f0cec516" containerName="storage-initializer"
Jul 31 15:04:26.454880 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.454859    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2e0c576-e7d0-45d7-842d-a890f0cec516" containerName="storage-initializer"
Jul 31 15:04:26.455023 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.454902    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2e0c576-e7d0-45d7-842d-a890f0cec516" containerName="storage-initializer"
Jul 31 15:04:26.455023 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.455005    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2e0c576-e7d0-45d7-842d-a890f0cec516" containerName="storage-initializer"
Jul 31 15:04:26.457853 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.457834    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"
Jul 31 15:04:26.460593 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.460575    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 31 15:04:26.460593 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.460587    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 31 15:04:26.468704 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.468684    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"]
Jul 31 15:04:26.564514 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.564475    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3846a890-d29b-448f-9b7e-d58ecd3a66c9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9n2td\" (UID: \"3846a890-d29b-448f-9b7e-d58ecd3a66c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"
Jul 31 15:04:26.564673 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.564540    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3846a890-d29b-448f-9b7e-d58ecd3a66c9-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9n2td\" (UID: \"3846a890-d29b-448f-9b7e-d58ecd3a66c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"
Jul 31 15:04:26.564673 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.564559    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzf9d\" (UniqueName: \"kubernetes.io/projected/3846a890-d29b-448f-9b7e-d58ecd3a66c9-kube-api-access-bzf9d\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9n2td\" (UID: \"3846a890-d29b-448f-9b7e-d58ecd3a66c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"
Jul 31 15:04:26.564673 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.564604    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3846a890-d29b-448f-9b7e-d58ecd3a66c9-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9n2td\" (UID: \"3846a890-d29b-448f-9b7e-d58ecd3a66c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"
Jul 31 15:04:26.564673 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.564628    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3846a890-d29b-448f-9b7e-d58ecd3a66c9-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9n2td\" (UID: \"3846a890-d29b-448f-9b7e-d58ecd3a66c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"
Jul 31 15:04:26.655369 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.655346    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q"
Jul 31 15:04:26.665566 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.665544    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3846a890-d29b-448f-9b7e-d58ecd3a66c9-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9n2td\" (UID: \"3846a890-d29b-448f-9b7e-d58ecd3a66c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"
Jul 31 15:04:26.665645 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.665575    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzf9d\" (UniqueName: \"kubernetes.io/projected/3846a890-d29b-448f-9b7e-d58ecd3a66c9-kube-api-access-bzf9d\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9n2td\" (UID: \"3846a890-d29b-448f-9b7e-d58ecd3a66c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"
Jul 31 15:04:26.665645 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.665601    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3846a890-d29b-448f-9b7e-d58ecd3a66c9-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9n2td\" (UID: \"3846a890-d29b-448f-9b7e-d58ecd3a66c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"
Jul 31 15:04:26.665645 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.665626    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3846a890-d29b-448f-9b7e-d58ecd3a66c9-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9n2td\" (UID: \"3846a890-d29b-448f-9b7e-d58ecd3a66c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"
Jul 31 15:04:26.665832 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.665656    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3846a890-d29b-448f-9b7e-d58ecd3a66c9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9n2td\" (UID: \"3846a890-d29b-448f-9b7e-d58ecd3a66c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"
Jul 31 15:04:26.665832 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:04:26.665810    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v1-predictor-serving-cert: secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 31 15:04:26.665935 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:04:26.665881    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3846a890-d29b-448f-9b7e-d58ecd3a66c9-proxy-tls podName:3846a890-d29b-448f-9b7e-d58ecd3a66c9 nodeName:}" failed. No retries permitted until 2026-07-31 15:04:27.165861248 +0000 UTC m=+963.477418847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3846a890-d29b-448f-9b7e-d58ecd3a66c9-proxy-tls") pod "isvc-autogluon-v1-predictor-574b5b7495-9n2td" (UID: "3846a890-d29b-448f-9b7e-d58ecd3a66c9") : secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 31 15:04:26.666057 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.666036    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3846a890-d29b-448f-9b7e-d58ecd3a66c9-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9n2td\" (UID: \"3846a890-d29b-448f-9b7e-d58ecd3a66c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"
Jul 31 15:04:26.666198 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.666180    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3846a890-d29b-448f-9b7e-d58ecd3a66c9-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9n2td\" (UID: \"3846a890-d29b-448f-9b7e-d58ecd3a66c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"
Jul 31 15:04:26.666279 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.666261    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3846a890-d29b-448f-9b7e-d58ecd3a66c9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9n2td\" (UID: \"3846a890-d29b-448f-9b7e-d58ecd3a66c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"
Jul 31 15:04:26.674600 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.674552    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzf9d\" (UniqueName: \"kubernetes.io/projected/3846a890-d29b-448f-9b7e-d58ecd3a66c9-kube-api-access-bzf9d\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9n2td\" (UID: \"3846a890-d29b-448f-9b7e-d58ecd3a66c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"
Jul 31 15:04:26.766708 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.766675    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srrzh\" (UniqueName: \"kubernetes.io/projected/b189726d-1b57-4ec4-876d-37850bb398d9-kube-api-access-srrzh\") pod \"b189726d-1b57-4ec4-876d-37850bb398d9\" (UID: \"b189726d-1b57-4ec4-876d-37850bb398d9\") "
Jul 31 15:04:26.766708 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.766713    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b189726d-1b57-4ec4-876d-37850bb398d9-kube-rbac-proxy-sar-config\") pod \"b189726d-1b57-4ec4-876d-37850bb398d9\" (UID: \"b189726d-1b57-4ec4-876d-37850bb398d9\") "
Jul 31 15:04:26.766956 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.766820    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b189726d-1b57-4ec4-876d-37850bb398d9-proxy-tls\") pod \"b189726d-1b57-4ec4-876d-37850bb398d9\" (UID: \"b189726d-1b57-4ec4-876d-37850bb398d9\") "
Jul 31 15:04:26.767138 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.767112    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b189726d-1b57-4ec4-876d-37850bb398d9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b189726d-1b57-4ec4-876d-37850bb398d9" (UID: "b189726d-1b57-4ec4-876d-37850bb398d9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:04:26.768972 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.768947    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b189726d-1b57-4ec4-876d-37850bb398d9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b189726d-1b57-4ec4-876d-37850bb398d9" (UID: "b189726d-1b57-4ec4-876d-37850bb398d9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:04:26.769060 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.769008    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b189726d-1b57-4ec4-876d-37850bb398d9-kube-api-access-srrzh" (OuterVolumeSpecName: "kube-api-access-srrzh") pod "b189726d-1b57-4ec4-876d-37850bb398d9" (UID: "b189726d-1b57-4ec4-876d-37850bb398d9"). InnerVolumeSpecName "kube-api-access-srrzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:04:26.857437 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.857408    2559 generic.go:356] "Generic (PLEG): container finished" podID="b189726d-1b57-4ec4-876d-37850bb398d9" containerID="3c1c163e996b4750ecf493a4231b8e62723587341dffcda0aaed2a2c8de72faf" exitCode=2
Jul 31 15:04:26.857437 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.857434    2559 generic.go:356] "Generic (PLEG): container finished" podID="b189726d-1b57-4ec4-876d-37850bb398d9" containerID="38b1661ddfdb2d39e050c0ba049f1828af9aceb003de7269fbaa8614d48ccf16" exitCode=2
Jul 31 15:04:26.857634 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.857449    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q"
Jul 31 15:04:26.857634 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.857483    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q" event={"ID":"b189726d-1b57-4ec4-876d-37850bb398d9","Type":"ContainerDied","Data":"3c1c163e996b4750ecf493a4231b8e62723587341dffcda0aaed2a2c8de72faf"}
Jul 31 15:04:26.857634 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.857578    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q" event={"ID":"b189726d-1b57-4ec4-876d-37850bb398d9","Type":"ContainerDied","Data":"38b1661ddfdb2d39e050c0ba049f1828af9aceb003de7269fbaa8614d48ccf16"}
Jul 31 15:04:26.857634 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.857612    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q" event={"ID":"b189726d-1b57-4ec4-876d-37850bb398d9","Type":"ContainerDied","Data":"6c366197063cf0392510f830d353af256c1275203d645eac2b872e1b6c241ced"}
Jul 31 15:04:26.857634 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.857621    2559 scope.go:117] "RemoveContainer" containerID="3c1c163e996b4750ecf493a4231b8e62723587341dffcda0aaed2a2c8de72faf"
Jul 31 15:04:26.860167 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.860145    2559 generic.go:356] "Generic (PLEG): container finished" podID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerID="13fa20bfa9dff46701c01f14cbe25956200feac408ba486e4b081c44538d4b80" exitCode=2
Jul 31 15:04:26.860283 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.860192    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" event={"ID":"4fc53b8a-3f86-47cf-af9c-96ab0b52b873","Type":"ContainerDied","Data":"13fa20bfa9dff46701c01f14cbe25956200feac408ba486e4b081c44538d4b80"}
Jul 31 15:04:26.866148 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.866134    2559 scope.go:117] "RemoveContainer" containerID="38b1661ddfdb2d39e050c0ba049f1828af9aceb003de7269fbaa8614d48ccf16"
Jul 31 15:04:26.867665 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.867648    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b189726d-1b57-4ec4-876d-37850bb398d9-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:04:26.867721 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.867668    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srrzh\" (UniqueName: \"kubernetes.io/projected/b189726d-1b57-4ec4-876d-37850bb398d9-kube-api-access-srrzh\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:04:26.867721 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.867679    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b189726d-1b57-4ec4-876d-37850bb398d9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:04:26.873855 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.873831    2559 scope.go:117] "RemoveContainer" containerID="3c1c163e996b4750ecf493a4231b8e62723587341dffcda0aaed2a2c8de72faf"
Jul 31 15:04:26.874133 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:04:26.874115    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c1c163e996b4750ecf493a4231b8e62723587341dffcda0aaed2a2c8de72faf\": container with ID starting with 3c1c163e996b4750ecf493a4231b8e62723587341dffcda0aaed2a2c8de72faf not found: ID does not exist" containerID="3c1c163e996b4750ecf493a4231b8e62723587341dffcda0aaed2a2c8de72faf"
Jul 31 15:04:26.874187 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.874143    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c1c163e996b4750ecf493a4231b8e62723587341dffcda0aaed2a2c8de72faf"} err="failed to get container status \"3c1c163e996b4750ecf493a4231b8e62723587341dffcda0aaed2a2c8de72faf\": rpc error: code = NotFound desc = could not find container \"3c1c163e996b4750ecf493a4231b8e62723587341dffcda0aaed2a2c8de72faf\": container with ID starting with 3c1c163e996b4750ecf493a4231b8e62723587341dffcda0aaed2a2c8de72faf not found: ID does not exist"
Jul 31 15:04:26.874187 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.874159    2559 scope.go:117] "RemoveContainer" containerID="38b1661ddfdb2d39e050c0ba049f1828af9aceb003de7269fbaa8614d48ccf16"
Jul 31 15:04:26.874383 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:04:26.874363    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38b1661ddfdb2d39e050c0ba049f1828af9aceb003de7269fbaa8614d48ccf16\": container with ID starting with 38b1661ddfdb2d39e050c0ba049f1828af9aceb003de7269fbaa8614d48ccf16 not found: ID does not exist" containerID="38b1661ddfdb2d39e050c0ba049f1828af9aceb003de7269fbaa8614d48ccf16"
Jul 31 15:04:26.874451 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.874392    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b1661ddfdb2d39e050c0ba049f1828af9aceb003de7269fbaa8614d48ccf16"} err="failed to get container status \"38b1661ddfdb2d39e050c0ba049f1828af9aceb003de7269fbaa8614d48ccf16\": rpc error: code = NotFound desc = could not find container \"38b1661ddfdb2d39e050c0ba049f1828af9aceb003de7269fbaa8614d48ccf16\": container with ID starting with 38b1661ddfdb2d39e050c0ba049f1828af9aceb003de7269fbaa8614d48ccf16 not found: ID does not exist"
Jul 31 15:04:26.874451 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.874413    2559 scope.go:117] "RemoveContainer" containerID="3c1c163e996b4750ecf493a4231b8e62723587341dffcda0aaed2a2c8de72faf"
Jul 31 15:04:26.874658 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.874639    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c1c163e996b4750ecf493a4231b8e62723587341dffcda0aaed2a2c8de72faf"} err="failed to get container status \"3c1c163e996b4750ecf493a4231b8e62723587341dffcda0aaed2a2c8de72faf\": rpc error: code = NotFound desc = could not find container \"3c1c163e996b4750ecf493a4231b8e62723587341dffcda0aaed2a2c8de72faf\": container with ID starting with 3c1c163e996b4750ecf493a4231b8e62723587341dffcda0aaed2a2c8de72faf not found: ID does not exist"
Jul 31 15:04:26.874709 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.874659    2559 scope.go:117] "RemoveContainer" containerID="38b1661ddfdb2d39e050c0ba049f1828af9aceb003de7269fbaa8614d48ccf16"
Jul 31 15:04:26.874887 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.874869    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b1661ddfdb2d39e050c0ba049f1828af9aceb003de7269fbaa8614d48ccf16"} err="failed to get container status \"38b1661ddfdb2d39e050c0ba049f1828af9aceb003de7269fbaa8614d48ccf16\": rpc error: code = NotFound desc = could not find container \"38b1661ddfdb2d39e050c0ba049f1828af9aceb003de7269fbaa8614d48ccf16\": container with ID starting with 38b1661ddfdb2d39e050c0ba049f1828af9aceb003de7269fbaa8614d48ccf16 not found: ID does not exist"
Jul 31 15:04:26.882247 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.882224    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q"]
Jul 31 15:04:26.885849 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:26.885830    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-kl99q"]
Jul 31 15:04:27.171446 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:27.171399    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3846a890-d29b-448f-9b7e-d58ecd3a66c9-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9n2td\" (UID: \"3846a890-d29b-448f-9b7e-d58ecd3a66c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"
Jul 31 15:04:27.171940 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:04:27.171556    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v1-predictor-serving-cert: secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 31 15:04:27.171940 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:04:27.171631    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3846a890-d29b-448f-9b7e-d58ecd3a66c9-proxy-tls podName:3846a890-d29b-448f-9b7e-d58ecd3a66c9 nodeName:}" failed. No retries permitted until 2026-07-31 15:04:28.171615559 +0000 UTC m=+964.483173159 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3846a890-d29b-448f-9b7e-d58ecd3a66c9-proxy-tls") pod "isvc-autogluon-v1-predictor-574b5b7495-9n2td" (UID: "3846a890-d29b-448f-9b7e-d58ecd3a66c9") : secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 31 15:04:28.180960 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:28.180905    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3846a890-d29b-448f-9b7e-d58ecd3a66c9-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9n2td\" (UID: \"3846a890-d29b-448f-9b7e-d58ecd3a66c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"
Jul 31 15:04:28.183443 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:28.183423    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3846a890-d29b-448f-9b7e-d58ecd3a66c9-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-9n2td\" (UID: \"3846a890-d29b-448f-9b7e-d58ecd3a66c9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"
Jul 31 15:04:28.221613 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:28.221588    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b189726d-1b57-4ec4-876d-37850bb398d9" path="/var/lib/kubelet/pods/b189726d-1b57-4ec4-876d-37850bb398d9/volumes"
Jul 31 15:04:28.294662 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:28.294630    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"
Jul 31 15:04:28.438663 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:28.438633    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"]
Jul 31 15:04:28.441068 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:04:28.441036    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3846a890_d29b_448f_9b7e_d58ecd3a66c9.slice/crio-f21f3b365f17cad917121a53485c3de3981cc2f988e8d7d8f08055c7bf558c2e WatchSource:0}: Error finding container f21f3b365f17cad917121a53485c3de3981cc2f988e8d7d8f08055c7bf558c2e: Status 404 returned error can't find the container with id f21f3b365f17cad917121a53485c3de3981cc2f988e8d7d8f08055c7bf558c2e
Jul 31 15:04:28.870912 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:28.870872    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td" event={"ID":"3846a890-d29b-448f-9b7e-d58ecd3a66c9","Type":"ContainerStarted","Data":"5b0ec52c25cf87704119d8590c53772629acdd28fa450929c5be712dcd6fd04a"}
Jul 31 15:04:28.871078 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:28.870944    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td" event={"ID":"3846a890-d29b-448f-9b7e-d58ecd3a66c9","Type":"ContainerStarted","Data":"f21f3b365f17cad917121a53485c3de3981cc2f988e8d7d8f08055c7bf558c2e"}
Jul 31 15:04:29.507006 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:29.506959    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 31 15:04:31.890361 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:31.890325    2559 generic.go:356] "Generic (PLEG): container finished" podID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerID="c5f12c8971762996101c631b85ad1e8dc65a4d51c676f5291a634eba8d3933b0" exitCode=0
Jul 31 15:04:31.890766 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:31.890396    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" event={"ID":"4fc53b8a-3f86-47cf-af9c-96ab0b52b873","Type":"ContainerDied","Data":"c5f12c8971762996101c631b85ad1e8dc65a4d51c676f5291a634eba8d3933b0"}
Jul 31 15:04:33.899658 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:33.899625    2559 generic.go:356] "Generic (PLEG): container finished" podID="3846a890-d29b-448f-9b7e-d58ecd3a66c9" containerID="5b0ec52c25cf87704119d8590c53772629acdd28fa450929c5be712dcd6fd04a" exitCode=0
Jul 31 15:04:33.900042 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:33.899701    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td" event={"ID":"3846a890-d29b-448f-9b7e-d58ecd3a66c9","Type":"ContainerDied","Data":"5b0ec52c25cf87704119d8590c53772629acdd28fa450929c5be712dcd6fd04a"}
Jul 31 15:04:34.507087 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:34.507033    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 31 15:04:34.511301 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:34.511231    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 31 15:04:34.512441 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:34.511812    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:04:39.506961 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:39.506909    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 31 15:04:39.507407 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:39.507064    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:04:44.506637 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:44.506593    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 31 15:04:44.511601 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:44.511566    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 31 15:04:44.512085 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:44.512027    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:04:49.506851 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:49.506723    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 31 15:04:54.506532 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:54.506480    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 31 15:04:54.511365 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:54.511319    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 31 15:04:54.511519 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:54.511459    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:04:54.511682 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:54.511654    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 15:04:54.511811 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:54.511795    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:04:56.602897 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:56.602864    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:04:56.717376 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:56.717292    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-kserve-provision-location\") pod \"4fc53b8a-3f86-47cf-af9c-96ab0b52b873\" (UID: \"4fc53b8a-3f86-47cf-af9c-96ab0b52b873\") "
Jul 31 15:04:56.717376 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:56.717328    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-cabundle-cert\") pod \"4fc53b8a-3f86-47cf-af9c-96ab0b52b873\" (UID: \"4fc53b8a-3f86-47cf-af9c-96ab0b52b873\") "
Jul 31 15:04:56.717376 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:56.717345    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-kube-rbac-proxy-sar-config\") pod \"4fc53b8a-3f86-47cf-af9c-96ab0b52b873\" (UID: \"4fc53b8a-3f86-47cf-af9c-96ab0b52b873\") "
Jul 31 15:04:56.717669 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:56.717400    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-proxy-tls\") pod \"4fc53b8a-3f86-47cf-af9c-96ab0b52b873\" (UID: \"4fc53b8a-3f86-47cf-af9c-96ab0b52b873\") "
Jul 31 15:04:56.717669 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:56.717457    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxvls\" (UniqueName: \"kubernetes.io/projected/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-kube-api-access-jxvls\") pod \"4fc53b8a-3f86-47cf-af9c-96ab0b52b873\" (UID: \"4fc53b8a-3f86-47cf-af9c-96ab0b52b873\") "
Jul 31 15:04:56.717809 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:56.717670    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4fc53b8a-3f86-47cf-af9c-96ab0b52b873" (UID: "4fc53b8a-3f86-47cf-af9c-96ab0b52b873"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:04:56.717809 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:56.717756    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4fc53b8a-3f86-47cf-af9c-96ab0b52b873" (UID: "4fc53b8a-3f86-47cf-af9c-96ab0b52b873"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:04:56.717809 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:56.717775    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4fc53b8a-3f86-47cf-af9c-96ab0b52b873" (UID: "4fc53b8a-3f86-47cf-af9c-96ab0b52b873"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:04:56.719945 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:56.719905    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-kube-api-access-jxvls" (OuterVolumeSpecName: "kube-api-access-jxvls") pod "4fc53b8a-3f86-47cf-af9c-96ab0b52b873" (UID: "4fc53b8a-3f86-47cf-af9c-96ab0b52b873"). InnerVolumeSpecName "kube-api-access-jxvls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:04:56.720053 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:56.719946    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4fc53b8a-3f86-47cf-af9c-96ab0b52b873" (UID: "4fc53b8a-3f86-47cf-af9c-96ab0b52b873"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:04:56.818638 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:56.818599    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxvls\" (UniqueName: \"kubernetes.io/projected/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-kube-api-access-jxvls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:04:56.818638 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:56.818632    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:04:56.818638 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:56.818642    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:04:56.818895 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:56.818651    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:04:56.818895 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:56.818661    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fc53b8a-3f86-47cf-af9c-96ab0b52b873-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:04:57.002772 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:57.002673    2559 generic.go:356] "Generic (PLEG): container finished" podID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerID="2cc9233491f69fac51af2640182c63f1572e39297d04252633c57765ee66414d" exitCode=0
Jul 31 15:04:57.002772 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:57.002765    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"
Jul 31 15:04:57.002997 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:57.002764    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" event={"ID":"4fc53b8a-3f86-47cf-af9c-96ab0b52b873","Type":"ContainerDied","Data":"2cc9233491f69fac51af2640182c63f1572e39297d04252633c57765ee66414d"}
Jul 31 15:04:57.002997 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:57.002845    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv" event={"ID":"4fc53b8a-3f86-47cf-af9c-96ab0b52b873","Type":"ContainerDied","Data":"5b0accbb04c2cb43579d7f76896fa603670e7eb78f33255e39ebf03668c65d2a"}
Jul 31 15:04:57.002997 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:57.002889    2559 scope.go:117] "RemoveContainer" containerID="2cc9233491f69fac51af2640182c63f1572e39297d04252633c57765ee66414d"
Jul 31 15:04:57.013086 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:57.013070    2559 scope.go:117] "RemoveContainer" containerID="13fa20bfa9dff46701c01f14cbe25956200feac408ba486e4b081c44538d4b80"
Jul 31 15:04:57.021920 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:57.021888    2559 scope.go:117] "RemoveContainer" containerID="c5f12c8971762996101c631b85ad1e8dc65a4d51c676f5291a634eba8d3933b0"
Jul 31 15:04:57.027408 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:57.027385    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"]
Jul 31 15:04:57.029955 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:57.029938    2559 scope.go:117] "RemoveContainer" containerID="82c5bdeed2ea2e2a0da040f4c9a18c86a7fad3d223f2c61a2dc6726473aec6bb"
Jul 31 15:04:57.031887 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:57.031868    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-754d978c8d-6gzmv"]
Jul 31 15:04:57.037324 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:57.037308    2559 scope.go:117] "RemoveContainer" containerID="2cc9233491f69fac51af2640182c63f1572e39297d04252633c57765ee66414d"
Jul 31 15:04:57.037647 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:04:57.037621    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cc9233491f69fac51af2640182c63f1572e39297d04252633c57765ee66414d\": container with ID starting with 2cc9233491f69fac51af2640182c63f1572e39297d04252633c57765ee66414d not found: ID does not exist" containerID="2cc9233491f69fac51af2640182c63f1572e39297d04252633c57765ee66414d"
Jul 31 15:04:57.037752 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:57.037659    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cc9233491f69fac51af2640182c63f1572e39297d04252633c57765ee66414d"} err="failed to get container status \"2cc9233491f69fac51af2640182c63f1572e39297d04252633c57765ee66414d\": rpc error: code = NotFound desc = could not find container \"2cc9233491f69fac51af2640182c63f1572e39297d04252633c57765ee66414d\": container with ID starting with 2cc9233491f69fac51af2640182c63f1572e39297d04252633c57765ee66414d not found: ID does not exist"
Jul 31 15:04:57.037752 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:57.037683    2559 scope.go:117] "RemoveContainer" containerID="13fa20bfa9dff46701c01f14cbe25956200feac408ba486e4b081c44538d4b80"
Jul 31 15:04:57.037994 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:04:57.037974    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13fa20bfa9dff46701c01f14cbe25956200feac408ba486e4b081c44538d4b80\": container with ID starting with 13fa20bfa9dff46701c01f14cbe25956200feac408ba486e4b081c44538d4b80 not found: ID does not exist" containerID="13fa20bfa9dff46701c01f14cbe25956200feac408ba486e4b081c44538d4b80"
Jul 31 15:04:57.038072 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:57.038002    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13fa20bfa9dff46701c01f14cbe25956200feac408ba486e4b081c44538d4b80"} err="failed to get container status \"13fa20bfa9dff46701c01f14cbe25956200feac408ba486e4b081c44538d4b80\": rpc error: code = NotFound desc = could not find container \"13fa20bfa9dff46701c01f14cbe25956200feac408ba486e4b081c44538d4b80\": container with ID starting with 13fa20bfa9dff46701c01f14cbe25956200feac408ba486e4b081c44538d4b80 not found: ID does not exist"
Jul 31 15:04:57.038072 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:57.038025    2559 scope.go:117] "RemoveContainer" containerID="c5f12c8971762996101c631b85ad1e8dc65a4d51c676f5291a634eba8d3933b0"
Jul 31 15:04:57.038283 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:04:57.038266    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5f12c8971762996101c631b85ad1e8dc65a4d51c676f5291a634eba8d3933b0\": container with ID starting with c5f12c8971762996101c631b85ad1e8dc65a4d51c676f5291a634eba8d3933b0 not found: ID does not exist" containerID="c5f12c8971762996101c631b85ad1e8dc65a4d51c676f5291a634eba8d3933b0"
Jul 31 15:04:57.038343 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:57.038287    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5f12c8971762996101c631b85ad1e8dc65a4d51c676f5291a634eba8d3933b0"} err="failed to get container status \"c5f12c8971762996101c631b85ad1e8dc65a4d51c676f5291a634eba8d3933b0\": rpc error: code = NotFound desc = could not find container \"c5f12c8971762996101c631b85ad1e8dc65a4d51c676f5291a634eba8d3933b0\": container with ID starting with c5f12c8971762996101c631b85ad1e8dc65a4d51c676f5291a634eba8d3933b0 not found: ID does not exist"
Jul 31 15:04:57.038343 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:57.038300    2559 scope.go:117] "RemoveContainer" containerID="82c5bdeed2ea2e2a0da040f4c9a18c86a7fad3d223f2c61a2dc6726473aec6bb"
Jul 31 15:04:57.038535 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:04:57.038520    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82c5bdeed2ea2e2a0da040f4c9a18c86a7fad3d223f2c61a2dc6726473aec6bb\": container with ID starting with 82c5bdeed2ea2e2a0da040f4c9a18c86a7fad3d223f2c61a2dc6726473aec6bb not found: ID does not exist" containerID="82c5bdeed2ea2e2a0da040f4c9a18c86a7fad3d223f2c61a2dc6726473aec6bb"
Jul 31 15:04:57.038583 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:57.038539    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82c5bdeed2ea2e2a0da040f4c9a18c86a7fad3d223f2c61a2dc6726473aec6bb"} err="failed to get container status \"82c5bdeed2ea2e2a0da040f4c9a18c86a7fad3d223f2c61a2dc6726473aec6bb\": rpc error: code = NotFound desc = could not find container \"82c5bdeed2ea2e2a0da040f4c9a18c86a7fad3d223f2c61a2dc6726473aec6bb\": container with ID starting with 82c5bdeed2ea2e2a0da040f4c9a18c86a7fad3d223f2c61a2dc6726473aec6bb not found: ID does not exist"
Jul 31 15:04:58.224478 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:04:58.222023    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" path="/var/lib/kubelet/pods/4fc53b8a-3f86-47cf-af9c-96ab0b52b873/volumes"
Jul 31 15:06:43.444195 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:06:43.444158    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td" event={"ID":"3846a890-d29b-448f-9b7e-d58ecd3a66c9","Type":"ContainerStarted","Data":"b094970b0c42585e9d2c65276ea06db7c844ee2d3007cd5cb0ae215e8d45c6c4"}
Jul 31 15:06:43.444873 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:06:43.444226    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td" event={"ID":"3846a890-d29b-448f-9b7e-d58ecd3a66c9","Type":"ContainerStarted","Data":"d96e92345daedc19df28bd43ec5a32ff5cdb990c67ef0e14e7b5ebe8ab4e5ca3"}
Jul 31 15:06:43.444873 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:06:43.444260    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"
Jul 31 15:06:43.444873 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:06:43.444391    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"
Jul 31 15:06:43.445667 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:06:43.445635    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td" podUID="3846a890-d29b-448f-9b7e-d58ecd3a66c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 31 15:06:43.467127 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:06:43.467068    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td" podStartSLOduration=3.527887366 podStartE2EDuration="2m17.467050917s" totalImagesPullingTime="2m8.916595268s" totalInitContainerRuntime="5.022568283s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:04:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:06:43.465801522 +0000 UTC m=+1099.777359144" watchObservedRunningTime="2026-07-31 15:06:43.467050917 +0000 UTC m=+1099.778608539"
Jul 31 15:06:44.448011 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:06:44.447973    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td" podUID="3846a890-d29b-448f-9b7e-d58ecd3a66c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 31 15:06:49.452907 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:06:49.452870    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"
Jul 31 15:06:49.453542 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:06:49.453500    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td" podUID="3846a890-d29b-448f-9b7e-d58ecd3a66c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 31 15:06:59.454488 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:06:59.454458    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"
Jul 31 15:07:12.909510 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:12.909472    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"]
Jul 31 15:07:12.909969 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:12.909783    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td" podUID="3846a890-d29b-448f-9b7e-d58ecd3a66c9" containerName="kserve-container" containerID="cri-o://d96e92345daedc19df28bd43ec5a32ff5cdb990c67ef0e14e7b5ebe8ab4e5ca3" gracePeriod=30
Jul 31 15:07:12.909969 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:12.909918    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td" podUID="3846a890-d29b-448f-9b7e-d58ecd3a66c9" containerName="kube-rbac-proxy" containerID="cri-o://b094970b0c42585e9d2c65276ea06db7c844ee2d3007cd5cb0ae215e8d45c6c4" gracePeriod=30
Jul 31 15:07:13.560503 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:13.560470    2559 generic.go:356] "Generic (PLEG): container finished" podID="3846a890-d29b-448f-9b7e-d58ecd3a66c9" containerID="b094970b0c42585e9d2c65276ea06db7c844ee2d3007cd5cb0ae215e8d45c6c4" exitCode=2
Jul 31 15:07:13.560690 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:13.560551    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td" event={"ID":"3846a890-d29b-448f-9b7e-d58ecd3a66c9","Type":"ContainerDied","Data":"b094970b0c42585e9d2c65276ea06db7c844ee2d3007cd5cb0ae215e8d45c6c4"}
Jul 31 15:07:14.374666 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.374634    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"
Jul 31 15:07:14.448187 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.448105    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3846a890-d29b-448f-9b7e-d58ecd3a66c9-kserve-provision-location\") pod \"3846a890-d29b-448f-9b7e-d58ecd3a66c9\" (UID: \"3846a890-d29b-448f-9b7e-d58ecd3a66c9\") "
Jul 31 15:07:14.448187 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.448161    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3846a890-d29b-448f-9b7e-d58ecd3a66c9-proxy-tls\") pod \"3846a890-d29b-448f-9b7e-d58ecd3a66c9\" (UID: \"3846a890-d29b-448f-9b7e-d58ecd3a66c9\") "
Jul 31 15:07:14.448187 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.448192    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3846a890-d29b-448f-9b7e-d58ecd3a66c9-cabundle-cert\") pod \"3846a890-d29b-448f-9b7e-d58ecd3a66c9\" (UID: \"3846a890-d29b-448f-9b7e-d58ecd3a66c9\") "
Jul 31 15:07:14.448494 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.448212    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzf9d\" (UniqueName: \"kubernetes.io/projected/3846a890-d29b-448f-9b7e-d58ecd3a66c9-kube-api-access-bzf9d\") pod \"3846a890-d29b-448f-9b7e-d58ecd3a66c9\" (UID: \"3846a890-d29b-448f-9b7e-d58ecd3a66c9\") "
Jul 31 15:07:14.448494 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.448260    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3846a890-d29b-448f-9b7e-d58ecd3a66c9-kube-rbac-proxy-sar-config\") pod \"3846a890-d29b-448f-9b7e-d58ecd3a66c9\" (UID: \"3846a890-d29b-448f-9b7e-d58ecd3a66c9\") "
Jul 31 15:07:14.448636 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.448603    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3846a890-d29b-448f-9b7e-d58ecd3a66c9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3846a890-d29b-448f-9b7e-d58ecd3a66c9" (UID: "3846a890-d29b-448f-9b7e-d58ecd3a66c9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:07:14.448764 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.448653    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3846a890-d29b-448f-9b7e-d58ecd3a66c9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3846a890-d29b-448f-9b7e-d58ecd3a66c9" (UID: "3846a890-d29b-448f-9b7e-d58ecd3a66c9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:07:14.450385 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.450359    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3846a890-d29b-448f-9b7e-d58ecd3a66c9-kube-api-access-bzf9d" (OuterVolumeSpecName: "kube-api-access-bzf9d") pod "3846a890-d29b-448f-9b7e-d58ecd3a66c9" (UID: "3846a890-d29b-448f-9b7e-d58ecd3a66c9"). InnerVolumeSpecName "kube-api-access-bzf9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:07:14.450781 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.450755    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3846a890-d29b-448f-9b7e-d58ecd3a66c9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3846a890-d29b-448f-9b7e-d58ecd3a66c9" (UID: "3846a890-d29b-448f-9b7e-d58ecd3a66c9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:07:14.454019 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.453997    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3846a890-d29b-448f-9b7e-d58ecd3a66c9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3846a890-d29b-448f-9b7e-d58ecd3a66c9" (UID: "3846a890-d29b-448f-9b7e-d58ecd3a66c9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:07:14.549118 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.549080    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3846a890-d29b-448f-9b7e-d58ecd3a66c9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:07:14.549118 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.549112    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3846a890-d29b-448f-9b7e-d58ecd3a66c9-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:07:14.549118 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.549123    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3846a890-d29b-448f-9b7e-d58ecd3a66c9-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:07:14.549382 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.549134    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3846a890-d29b-448f-9b7e-d58ecd3a66c9-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:07:14.549382 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.549142    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzf9d\" (UniqueName: \"kubernetes.io/projected/3846a890-d29b-448f-9b7e-d58ecd3a66c9-kube-api-access-bzf9d\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:07:14.571387 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.571349    2559 generic.go:356] "Generic (PLEG): container finished" podID="3846a890-d29b-448f-9b7e-d58ecd3a66c9" containerID="d96e92345daedc19df28bd43ec5a32ff5cdb990c67ef0e14e7b5ebe8ab4e5ca3" exitCode=0
Jul 31 15:07:14.571541 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.571427    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"
Jul 31 15:07:14.571541 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.571434    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td" event={"ID":"3846a890-d29b-448f-9b7e-d58ecd3a66c9","Type":"ContainerDied","Data":"d96e92345daedc19df28bd43ec5a32ff5cdb990c67ef0e14e7b5ebe8ab4e5ca3"}
Jul 31 15:07:14.571541 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.571538    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td" event={"ID":"3846a890-d29b-448f-9b7e-d58ecd3a66c9","Type":"ContainerDied","Data":"f21f3b365f17cad917121a53485c3de3981cc2f988e8d7d8f08055c7bf558c2e"}
Jul 31 15:07:14.571645 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.571574    2559 scope.go:117] "RemoveContainer" containerID="b094970b0c42585e9d2c65276ea06db7c844ee2d3007cd5cb0ae215e8d45c6c4"
Jul 31 15:07:14.580471 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.580452    2559 scope.go:117] "RemoveContainer" containerID="d96e92345daedc19df28bd43ec5a32ff5cdb990c67ef0e14e7b5ebe8ab4e5ca3"
Jul 31 15:07:14.587687 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.587670    2559 scope.go:117] "RemoveContainer" containerID="5b0ec52c25cf87704119d8590c53772629acdd28fa450929c5be712dcd6fd04a"
Jul 31 15:07:14.594403 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.594381    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"]
Jul 31 15:07:14.595551 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.595537    2559 scope.go:117] "RemoveContainer" containerID="b094970b0c42585e9d2c65276ea06db7c844ee2d3007cd5cb0ae215e8d45c6c4"
Jul 31 15:07:14.596017 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:07:14.595981    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b094970b0c42585e9d2c65276ea06db7c844ee2d3007cd5cb0ae215e8d45c6c4\": container with ID starting with b094970b0c42585e9d2c65276ea06db7c844ee2d3007cd5cb0ae215e8d45c6c4 not found: ID does not exist" containerID="b094970b0c42585e9d2c65276ea06db7c844ee2d3007cd5cb0ae215e8d45c6c4"
Jul 31 15:07:14.596120 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.596015    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b094970b0c42585e9d2c65276ea06db7c844ee2d3007cd5cb0ae215e8d45c6c4"} err="failed to get container status \"b094970b0c42585e9d2c65276ea06db7c844ee2d3007cd5cb0ae215e8d45c6c4\": rpc error: code = NotFound desc = could not find container \"b094970b0c42585e9d2c65276ea06db7c844ee2d3007cd5cb0ae215e8d45c6c4\": container with ID starting with b094970b0c42585e9d2c65276ea06db7c844ee2d3007cd5cb0ae215e8d45c6c4 not found: ID does not exist"
Jul 31 15:07:14.596120 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.596041    2559 scope.go:117] "RemoveContainer" containerID="d96e92345daedc19df28bd43ec5a32ff5cdb990c67ef0e14e7b5ebe8ab4e5ca3"
Jul 31 15:07:14.596482 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:07:14.596450    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d96e92345daedc19df28bd43ec5a32ff5cdb990c67ef0e14e7b5ebe8ab4e5ca3\": container with ID starting with d96e92345daedc19df28bd43ec5a32ff5cdb990c67ef0e14e7b5ebe8ab4e5ca3 not found: ID does not exist" containerID="d96e92345daedc19df28bd43ec5a32ff5cdb990c67ef0e14e7b5ebe8ab4e5ca3"
Jul 31 15:07:14.596627 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.596492    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d96e92345daedc19df28bd43ec5a32ff5cdb990c67ef0e14e7b5ebe8ab4e5ca3"} err="failed to get container status \"d96e92345daedc19df28bd43ec5a32ff5cdb990c67ef0e14e7b5ebe8ab4e5ca3\": rpc error: code = NotFound desc = could not find container \"d96e92345daedc19df28bd43ec5a32ff5cdb990c67ef0e14e7b5ebe8ab4e5ca3\": container with ID starting with d96e92345daedc19df28bd43ec5a32ff5cdb990c67ef0e14e7b5ebe8ab4e5ca3 not found: ID does not exist"
Jul 31 15:07:14.596627 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.596517    2559 scope.go:117] "RemoveContainer" containerID="5b0ec52c25cf87704119d8590c53772629acdd28fa450929c5be712dcd6fd04a"
Jul 31 15:07:14.596819 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:07:14.596799    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b0ec52c25cf87704119d8590c53772629acdd28fa450929c5be712dcd6fd04a\": container with ID starting with 5b0ec52c25cf87704119d8590c53772629acdd28fa450929c5be712dcd6fd04a not found: ID does not exist" containerID="5b0ec52c25cf87704119d8590c53772629acdd28fa450929c5be712dcd6fd04a"
Jul 31 15:07:14.596868 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.596827    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b0ec52c25cf87704119d8590c53772629acdd28fa450929c5be712dcd6fd04a"} err="failed to get container status \"5b0ec52c25cf87704119d8590c53772629acdd28fa450929c5be712dcd6fd04a\": rpc error: code = NotFound desc = could not find container \"5b0ec52c25cf87704119d8590c53772629acdd28fa450929c5be712dcd6fd04a\": container with ID starting with 5b0ec52c25cf87704119d8590c53772629acdd28fa450929c5be712dcd6fd04a not found: ID does not exist"
Jul 31 15:07:14.597826 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:14.597811    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-9n2td"]
Jul 31 15:07:16.220367 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:16.220336    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3846a890-d29b-448f-9b7e-d58ecd3a66c9" path="/var/lib/kubelet/pods/3846a890-d29b-448f-9b7e-d58ecd3a66c9/volumes"
Jul 31 15:07:18.091806 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.091772    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"]
Jul 31 15:07:18.092193 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.092032    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="kube-rbac-proxy"
Jul 31 15:07:18.092193 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.092044    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="kube-rbac-proxy"
Jul 31 15:07:18.092193 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.092053    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="agent"
Jul 31 15:07:18.092193 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.092059    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="agent"
Jul 31 15:07:18.092193 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.092068    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b189726d-1b57-4ec4-876d-37850bb398d9" containerName="kube-rbac-proxy"
Jul 31 15:07:18.092193 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.092074    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b189726d-1b57-4ec4-876d-37850bb398d9" containerName="kube-rbac-proxy"
Jul 31 15:07:18.092193 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.092079    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3846a890-d29b-448f-9b7e-d58ecd3a66c9" containerName="kserve-container"
Jul 31 15:07:18.092193 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.092084    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3846a890-d29b-448f-9b7e-d58ecd3a66c9" containerName="kserve-container"
Jul 31 15:07:18.092193 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.092092    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b189726d-1b57-4ec4-876d-37850bb398d9" containerName="kserve-container"
Jul 31 15:07:18.092193 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.092097    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b189726d-1b57-4ec4-876d-37850bb398d9" containerName="kserve-container"
Jul 31 15:07:18.092193 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.092107    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3846a890-d29b-448f-9b7e-d58ecd3a66c9" containerName="kube-rbac-proxy"
Jul 31 15:07:18.092193 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.092112    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3846a890-d29b-448f-9b7e-d58ecd3a66c9" containerName="kube-rbac-proxy"
Jul 31 15:07:18.092193 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.092121    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3846a890-d29b-448f-9b7e-d58ecd3a66c9" containerName="storage-initializer"
Jul 31 15:07:18.092193 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.092126    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3846a890-d29b-448f-9b7e-d58ecd3a66c9" containerName="storage-initializer"
Jul 31 15:07:18.092193 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.092131    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="kserve-container"
Jul 31 15:07:18.092193 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.092136    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="kserve-container"
Jul 31 15:07:18.092193 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.092142    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="storage-initializer"
Jul 31 15:07:18.092193 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.092146    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="storage-initializer"
Jul 31 15:07:18.092193 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.092188    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3846a890-d29b-448f-9b7e-d58ecd3a66c9" containerName="kserve-container"
Jul 31 15:07:18.092193 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.092197    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="agent"
Jul 31 15:07:18.092193 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.092203    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b189726d-1b57-4ec4-876d-37850bb398d9" containerName="kube-rbac-proxy"
Jul 31 15:07:18.092830 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.092209    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b189726d-1b57-4ec4-876d-37850bb398d9" containerName="kserve-container"
Jul 31 15:07:18.092830 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.092215    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="kserve-container"
Jul 31 15:07:18.092830 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.092221    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fc53b8a-3f86-47cf-af9c-96ab0b52b873" containerName="kube-rbac-proxy"
Jul 31 15:07:18.092830 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.092227    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3846a890-d29b-448f-9b7e-d58ecd3a66c9" containerName="kube-rbac-proxy"
Jul 31 15:07:18.130465 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.130430    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"]
Jul 31 15:07:18.130645 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.130580    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"
Jul 31 15:07:18.133788 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.133767    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 15:07:18.134082 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.134065    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 31 15:07:18.134179 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.134131    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 31 15:07:18.134179 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.134156    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:07:18.134179 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.134170    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 15:07:18.134714 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.134696    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dd67w\""
Jul 31 15:07:18.134838 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.134779    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 15:07:18.274431 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.274403    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3603558-f989-466b-9522-273ca41fdd21-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-qgvt5\" (UID: \"c3603558-f989-466b-9522-273ca41fdd21\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"
Jul 31 15:07:18.274603 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.274441    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3603558-f989-466b-9522-273ca41fdd21-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-qgvt5\" (UID: \"c3603558-f989-466b-9522-273ca41fdd21\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"
Jul 31 15:07:18.274603 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.274462    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wssv\" (UniqueName: \"kubernetes.io/projected/c3603558-f989-466b-9522-273ca41fdd21-kube-api-access-8wssv\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-qgvt5\" (UID: \"c3603558-f989-466b-9522-273ca41fdd21\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"
Jul 31 15:07:18.274603 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.274526    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3603558-f989-466b-9522-273ca41fdd21-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-qgvt5\" (UID: \"c3603558-f989-466b-9522-273ca41fdd21\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"
Jul 31 15:07:18.274603 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.274580    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3603558-f989-466b-9522-273ca41fdd21-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-qgvt5\" (UID: \"c3603558-f989-466b-9522-273ca41fdd21\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"
Jul 31 15:07:18.375431 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.375402    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3603558-f989-466b-9522-273ca41fdd21-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-qgvt5\" (UID: \"c3603558-f989-466b-9522-273ca41fdd21\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"
Jul 31 15:07:18.375629 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.375441    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3603558-f989-466b-9522-273ca41fdd21-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-qgvt5\" (UID: \"c3603558-f989-466b-9522-273ca41fdd21\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"
Jul 31 15:07:18.375629 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.375466    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wssv\" (UniqueName: \"kubernetes.io/projected/c3603558-f989-466b-9522-273ca41fdd21-kube-api-access-8wssv\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-qgvt5\" (UID: \"c3603558-f989-466b-9522-273ca41fdd21\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"
Jul 31 15:07:18.375629 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.375499    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3603558-f989-466b-9522-273ca41fdd21-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-qgvt5\" (UID: \"c3603558-f989-466b-9522-273ca41fdd21\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"
Jul 31 15:07:18.375629 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.375544    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3603558-f989-466b-9522-273ca41fdd21-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-qgvt5\" (UID: \"c3603558-f989-466b-9522-273ca41fdd21\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"
Jul 31 15:07:18.376157 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.376130    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3603558-f989-466b-9522-273ca41fdd21-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-qgvt5\" (UID: \"c3603558-f989-466b-9522-273ca41fdd21\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"
Jul 31 15:07:18.376299 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.376284    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3603558-f989-466b-9522-273ca41fdd21-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-qgvt5\" (UID: \"c3603558-f989-466b-9522-273ca41fdd21\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"
Jul 31 15:07:18.376344 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.376307    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3603558-f989-466b-9522-273ca41fdd21-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-qgvt5\" (UID: \"c3603558-f989-466b-9522-273ca41fdd21\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"
Jul 31 15:07:18.378170 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.378149    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3603558-f989-466b-9522-273ca41fdd21-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-qgvt5\" (UID: \"c3603558-f989-466b-9522-273ca41fdd21\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"
Jul 31 15:07:18.386659 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.386637    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wssv\" (UniqueName: \"kubernetes.io/projected/c3603558-f989-466b-9522-273ca41fdd21-kube-api-access-8wssv\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-qgvt5\" (UID: \"c3603558-f989-466b-9522-273ca41fdd21\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"
Jul 31 15:07:18.441356 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.441314    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"
Jul 31 15:07:18.593630 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:18.593526    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"]
Jul 31 15:07:18.596258 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:07:18.596229    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3603558_f989_466b_9522_273ca41fdd21.slice/crio-031d75f240cbb5fb9d0488979945f8f77b0c71c65f5ae78e8bed2a3554b11098 WatchSource:0}: Error finding container 031d75f240cbb5fb9d0488979945f8f77b0c71c65f5ae78e8bed2a3554b11098: Status 404 returned error can't find the container with id 031d75f240cbb5fb9d0488979945f8f77b0c71c65f5ae78e8bed2a3554b11098
Jul 31 15:07:19.591291 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:19.591243    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5" event={"ID":"c3603558-f989-466b-9522-273ca41fdd21","Type":"ContainerStarted","Data":"c4fc65f6b67aa552ff81a62cc273bfacd289519c67c7c95e202d74db0127f3e4"}
Jul 31 15:07:19.591692 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:19.591322    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5" event={"ID":"c3603558-f989-466b-9522-273ca41fdd21","Type":"ContainerStarted","Data":"031d75f240cbb5fb9d0488979945f8f77b0c71c65f5ae78e8bed2a3554b11098"}
Jul 31 15:07:24.610789 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:24.610760    2559 generic.go:356] "Generic (PLEG): container finished" podID="c3603558-f989-466b-9522-273ca41fdd21" containerID="c4fc65f6b67aa552ff81a62cc273bfacd289519c67c7c95e202d74db0127f3e4" exitCode=0
Jul 31 15:07:24.611247 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:24.610840    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5" event={"ID":"c3603558-f989-466b-9522-273ca41fdd21","Type":"ContainerDied","Data":"c4fc65f6b67aa552ff81a62cc273bfacd289519c67c7c95e202d74db0127f3e4"}
Jul 31 15:07:25.618052 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:25.618014    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5" event={"ID":"c3603558-f989-466b-9522-273ca41fdd21","Type":"ContainerStarted","Data":"36792756724ab556289ab2188604658bdc53db8e97a455732dcce35b132fe198"}
Jul 31 15:07:25.618533 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:25.618089    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5" event={"ID":"c3603558-f989-466b-9522-273ca41fdd21","Type":"ContainerStarted","Data":"035a399c0e2ff8127dd1e0da7ab87cf217da16846c69aca861b2b26e375d239d"}
Jul 31 15:07:25.618533 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:25.618305    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"
Jul 31 15:07:25.639671 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:25.639625    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5" podStartSLOduration=2.6329683680000002 podStartE2EDuration="7.639609059s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.006640691s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:07:25.638385332 +0000 UTC m=+1141.949942955" watchObservedRunningTime="2026-07-31 15:07:25.639609059 +0000 UTC m=+1141.951166680"
Jul 31 15:07:26.622206 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:26.622167    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"
Jul 31 15:07:32.633027 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:07:32.632998    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"
Jul 31 15:08:57.220547 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:08:57.220509    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"
Jul 31 15:09:04.458946 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:04.458910    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"]
Jul 31 15:09:04.459319 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:04.459129    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5" podUID="c3603558-f989-466b-9522-273ca41fdd21" containerName="kserve-container" containerID="cri-o://035a399c0e2ff8127dd1e0da7ab87cf217da16846c69aca861b2b26e375d239d" gracePeriod=30
Jul 31 15:09:04.459319 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:04.459162    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5" podUID="c3603558-f989-466b-9522-273ca41fdd21" containerName="kube-rbac-proxy" containerID="cri-o://36792756724ab556289ab2188604658bdc53db8e97a455732dcce35b132fe198" gracePeriod=30
Jul 31 15:09:04.976700 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:04.976669    2559 generic.go:356] "Generic (PLEG): container finished" podID="c3603558-f989-466b-9522-273ca41fdd21" containerID="36792756724ab556289ab2188604658bdc53db8e97a455732dcce35b132fe198" exitCode=2
Jul 31 15:09:04.976921 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:04.976766    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5" event={"ID":"c3603558-f989-466b-9522-273ca41fdd21","Type":"ContainerDied","Data":"36792756724ab556289ab2188604658bdc53db8e97a455732dcce35b132fe198"}
Jul 31 15:09:05.895811 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:05.895786    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"
Jul 31 15:09:05.987207 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:05.987121    2559 generic.go:356] "Generic (PLEG): container finished" podID="c3603558-f989-466b-9522-273ca41fdd21" containerID="035a399c0e2ff8127dd1e0da7ab87cf217da16846c69aca861b2b26e375d239d" exitCode=0
Jul 31 15:09:05.987207 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:05.987160    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5" event={"ID":"c3603558-f989-466b-9522-273ca41fdd21","Type":"ContainerDied","Data":"035a399c0e2ff8127dd1e0da7ab87cf217da16846c69aca861b2b26e375d239d"}
Jul 31 15:09:05.987207 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:05.987196    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"
Jul 31 15:09:05.987445 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:05.987238    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5" event={"ID":"c3603558-f989-466b-9522-273ca41fdd21","Type":"ContainerDied","Data":"031d75f240cbb5fb9d0488979945f8f77b0c71c65f5ae78e8bed2a3554b11098"}
Jul 31 15:09:05.987445 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:05.987268    2559 scope.go:117] "RemoveContainer" containerID="36792756724ab556289ab2188604658bdc53db8e97a455732dcce35b132fe198"
Jul 31 15:09:05.991237 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:05.991217    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3603558-f989-466b-9522-273ca41fdd21-cabundle-cert\") pod \"c3603558-f989-466b-9522-273ca41fdd21\" (UID: \"c3603558-f989-466b-9522-273ca41fdd21\") "
Jul 31 15:09:05.991354 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:05.991269    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3603558-f989-466b-9522-273ca41fdd21-proxy-tls\") pod \"c3603558-f989-466b-9522-273ca41fdd21\" (UID: \"c3603558-f989-466b-9522-273ca41fdd21\") "
Jul 31 15:09:05.991354 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:05.991304    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3603558-f989-466b-9522-273ca41fdd21-kube-rbac-proxy-sar-config\") pod \"c3603558-f989-466b-9522-273ca41fdd21\" (UID: \"c3603558-f989-466b-9522-273ca41fdd21\") "
Jul 31 15:09:05.991354 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:05.991329    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3603558-f989-466b-9522-273ca41fdd21-kserve-provision-location\") pod \"c3603558-f989-466b-9522-273ca41fdd21\" (UID: \"c3603558-f989-466b-9522-273ca41fdd21\") "
Jul 31 15:09:05.991354 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:05.991347    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wssv\" (UniqueName: \"kubernetes.io/projected/c3603558-f989-466b-9522-273ca41fdd21-kube-api-access-8wssv\") pod \"c3603558-f989-466b-9522-273ca41fdd21\" (UID: \"c3603558-f989-466b-9522-273ca41fdd21\") "
Jul 31 15:09:05.991619 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:05.991573    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3603558-f989-466b-9522-273ca41fdd21-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c3603558-f989-466b-9522-273ca41fdd21" (UID: "c3603558-f989-466b-9522-273ca41fdd21"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:09:05.991876 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:05.991849    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3603558-f989-466b-9522-273ca41fdd21-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c3603558-f989-466b-9522-273ca41fdd21" (UID: "c3603558-f989-466b-9522-273ca41fdd21"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:09:05.993652 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:05.993604    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3603558-f989-466b-9522-273ca41fdd21-kube-api-access-8wssv" (OuterVolumeSpecName: "kube-api-access-8wssv") pod "c3603558-f989-466b-9522-273ca41fdd21" (UID: "c3603558-f989-466b-9522-273ca41fdd21"). InnerVolumeSpecName "kube-api-access-8wssv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:09:05.994037 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:05.994010    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3603558-f989-466b-9522-273ca41fdd21-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c3603558-f989-466b-9522-273ca41fdd21" (UID: "c3603558-f989-466b-9522-273ca41fdd21"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:09:05.995900 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:05.995869    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3603558-f989-466b-9522-273ca41fdd21-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c3603558-f989-466b-9522-273ca41fdd21" (UID: "c3603558-f989-466b-9522-273ca41fdd21"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:09:05.996696 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:05.996671    2559 scope.go:117] "RemoveContainer" containerID="035a399c0e2ff8127dd1e0da7ab87cf217da16846c69aca861b2b26e375d239d"
Jul 31 15:09:06.008838 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:06.008816    2559 scope.go:117] "RemoveContainer" containerID="c4fc65f6b67aa552ff81a62cc273bfacd289519c67c7c95e202d74db0127f3e4"
Jul 31 15:09:06.018002 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:06.017982    2559 scope.go:117] "RemoveContainer" containerID="36792756724ab556289ab2188604658bdc53db8e97a455732dcce35b132fe198"
Jul 31 15:09:06.018353 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:09:06.018330    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36792756724ab556289ab2188604658bdc53db8e97a455732dcce35b132fe198\": container with ID starting with 36792756724ab556289ab2188604658bdc53db8e97a455732dcce35b132fe198 not found: ID does not exist" containerID="36792756724ab556289ab2188604658bdc53db8e97a455732dcce35b132fe198"
Jul 31 15:09:06.018433 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:06.018366    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36792756724ab556289ab2188604658bdc53db8e97a455732dcce35b132fe198"} err="failed to get container status \"36792756724ab556289ab2188604658bdc53db8e97a455732dcce35b132fe198\": rpc error: code = NotFound desc = could not find container \"36792756724ab556289ab2188604658bdc53db8e97a455732dcce35b132fe198\": container with ID starting with 36792756724ab556289ab2188604658bdc53db8e97a455732dcce35b132fe198 not found: ID does not exist"
Jul 31 15:09:06.018433 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:06.018392    2559 scope.go:117] "RemoveContainer" containerID="035a399c0e2ff8127dd1e0da7ab87cf217da16846c69aca861b2b26e375d239d"
Jul 31 15:09:06.018702 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:09:06.018675    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"035a399c0e2ff8127dd1e0da7ab87cf217da16846c69aca861b2b26e375d239d\": container with ID starting with 035a399c0e2ff8127dd1e0da7ab87cf217da16846c69aca861b2b26e375d239d not found: ID does not exist" containerID="035a399c0e2ff8127dd1e0da7ab87cf217da16846c69aca861b2b26e375d239d"
Jul 31 15:09:06.018774 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:06.018702    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"035a399c0e2ff8127dd1e0da7ab87cf217da16846c69aca861b2b26e375d239d"} err="failed to get container status \"035a399c0e2ff8127dd1e0da7ab87cf217da16846c69aca861b2b26e375d239d\": rpc error: code = NotFound desc = could not find container \"035a399c0e2ff8127dd1e0da7ab87cf217da16846c69aca861b2b26e375d239d\": container with ID starting with 035a399c0e2ff8127dd1e0da7ab87cf217da16846c69aca861b2b26e375d239d not found: ID does not exist"
Jul 31 15:09:06.018774 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:06.018721    2559 scope.go:117] "RemoveContainer" containerID="c4fc65f6b67aa552ff81a62cc273bfacd289519c67c7c95e202d74db0127f3e4"
Jul 31 15:09:06.019015 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:09:06.018996    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4fc65f6b67aa552ff81a62cc273bfacd289519c67c7c95e202d74db0127f3e4\": container with ID starting with c4fc65f6b67aa552ff81a62cc273bfacd289519c67c7c95e202d74db0127f3e4 not found: ID does not exist" containerID="c4fc65f6b67aa552ff81a62cc273bfacd289519c67c7c95e202d74db0127f3e4"
Jul 31 15:09:06.019096 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:06.019021    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4fc65f6b67aa552ff81a62cc273bfacd289519c67c7c95e202d74db0127f3e4"} err="failed to get container status \"c4fc65f6b67aa552ff81a62cc273bfacd289519c67c7c95e202d74db0127f3e4\": rpc error: code = NotFound desc = could not find container \"c4fc65f6b67aa552ff81a62cc273bfacd289519c67c7c95e202d74db0127f3e4\": container with ID starting with c4fc65f6b67aa552ff81a62cc273bfacd289519c67c7c95e202d74db0127f3e4 not found: ID does not exist"
Jul 31 15:09:06.092222 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:06.092187    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3603558-f989-466b-9522-273ca41fdd21-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:09:06.092222 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:06.092217    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3603558-f989-466b-9522-273ca41fdd21-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:09:06.092222 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:06.092227    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3603558-f989-466b-9522-273ca41fdd21-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:09:06.092453 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:06.092238    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wssv\" (UniqueName: \"kubernetes.io/projected/c3603558-f989-466b-9522-273ca41fdd21-kube-api-access-8wssv\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:09:06.092453 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:06.092249    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3603558-f989-466b-9522-273ca41fdd21-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:09:06.306926 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:06.306854    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"]
Jul 31 15:09:06.312668 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:06.312645    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-qgvt5"]
Jul 31 15:09:08.220442 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:08.220404    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3603558-f989-466b-9522-273ca41fdd21" path="/var/lib/kubelet/pods/c3603558-f989-466b-9522-273ca41fdd21/volumes"
Jul 31 15:09:09.628519 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.628485    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"]
Jul 31 15:09:09.628897 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.628782    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3603558-f989-466b-9522-273ca41fdd21" containerName="storage-initializer"
Jul 31 15:09:09.628897 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.628795    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3603558-f989-466b-9522-273ca41fdd21" containerName="storage-initializer"
Jul 31 15:09:09.628897 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.628803    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3603558-f989-466b-9522-273ca41fdd21" containerName="kube-rbac-proxy"
Jul 31 15:09:09.628897 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.628809    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3603558-f989-466b-9522-273ca41fdd21" containerName="kube-rbac-proxy"
Jul 31 15:09:09.628897 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.628820    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3603558-f989-466b-9522-273ca41fdd21" containerName="kserve-container"
Jul 31 15:09:09.628897 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.628826    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3603558-f989-466b-9522-273ca41fdd21" containerName="kserve-container"
Jul 31 15:09:09.628897 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.628869    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3603558-f989-466b-9522-273ca41fdd21" containerName="kserve-container"
Jul 31 15:09:09.628897 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.628880    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3603558-f989-466b-9522-273ca41fdd21" containerName="kube-rbac-proxy"
Jul 31 15:09:09.632290 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.632271    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"
Jul 31 15:09:09.640641 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.640619    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:09:09.640763 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.640643    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 15:09:09.641622 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.641603    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 15:09:09.641748 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.641642    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 15:09:09.642123 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.642108    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 31 15:09:09.642213 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.642127    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dd67w\""
Jul 31 15:09:09.642363 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.642348    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 31 15:09:09.647324 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.647305    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"]
Jul 31 15:09:09.718753 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.718700    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qj7n\" (UniqueName: \"kubernetes.io/projected/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-kube-api-access-8qj7n\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd\" (UID: \"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"
Jul 31 15:09:09.718925 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.718769    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd\" (UID: \"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"
Jul 31 15:09:09.718925 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.718820    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd\" (UID: \"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"
Jul 31 15:09:09.718925 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.718854    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd\" (UID: \"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"
Jul 31 15:09:09.718925 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.718875    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd\" (UID: \"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"
Jul 31 15:09:09.820015 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.819981    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qj7n\" (UniqueName: \"kubernetes.io/projected/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-kube-api-access-8qj7n\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd\" (UID: \"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"
Jul 31 15:09:09.820015 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.820022    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd\" (UID: \"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"
Jul 31 15:09:09.820274 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.820045    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd\" (UID: \"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"
Jul 31 15:09:09.820274 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.820072    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd\" (UID: \"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"
Jul 31 15:09:09.820274 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.820099    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd\" (UID: \"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"
Jul 31 15:09:09.820489 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.820467    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd\" (UID: \"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"
Jul 31 15:09:09.820887 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.820867    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd\" (UID: \"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"
Jul 31 15:09:09.820938 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.820867    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd\" (UID: \"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"
Jul 31 15:09:09.822713 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.822696    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd\" (UID: \"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"
Jul 31 15:09:09.830322 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.830300    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qj7n\" (UniqueName: \"kubernetes.io/projected/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-kube-api-access-8qj7n\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd\" (UID: \"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"
Jul 31 15:09:09.941145 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:09.941056    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"
Jul 31 15:09:10.075621 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:10.075596    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"]
Jul 31 15:09:10.078098 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:09:10.078075    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac3b1344_c4e2_4d86_ab8c_3fc7179843c3.slice/crio-d609f38601e5bd12015ba29309e6823ace735d55e3d80e744062fb658e06f8be WatchSource:0}: Error finding container d609f38601e5bd12015ba29309e6823ace735d55e3d80e744062fb658e06f8be: Status 404 returned error can't find the container with id d609f38601e5bd12015ba29309e6823ace735d55e3d80e744062fb658e06f8be
Jul 31 15:09:11.007923 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:11.007880    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd" event={"ID":"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3","Type":"ContainerStarted","Data":"bbc8f7c78c3459f24224b8461ff173e3186296dd5e3d7d8a3e22336565ac08f2"}
Jul 31 15:09:11.008346 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:11.007946    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd" event={"ID":"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3","Type":"ContainerStarted","Data":"d609f38601e5bd12015ba29309e6823ace735d55e3d80e744062fb658e06f8be"}
Jul 31 15:09:39.107417 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:39.107382    2559 generic.go:356] "Generic (PLEG): container finished" podID="ac3b1344-c4e2-4d86-ab8c-3fc7179843c3" containerID="bbc8f7c78c3459f24224b8461ff173e3186296dd5e3d7d8a3e22336565ac08f2" exitCode=0
Jul 31 15:09:39.107876 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:39.107453    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd" event={"ID":"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3","Type":"ContainerDied","Data":"bbc8f7c78c3459f24224b8461ff173e3186296dd5e3d7d8a3e22336565ac08f2"}
Jul 31 15:09:40.113017 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:40.112981    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd" event={"ID":"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3","Type":"ContainerStarted","Data":"96ec58dc852c5dbe5a93294963c6a279b88197667c63705fd679d4a101865a56"}
Jul 31 15:09:40.113449 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:40.113053    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd" event={"ID":"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3","Type":"ContainerStarted","Data":"81cdbb61613b7266e50ac1fdaa7bc7a9c253578ac86585c82187266a620d36a9"}
Jul 31 15:09:40.113449 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:40.113272    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"
Jul 31 15:09:40.135074 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:40.135019    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd" podStartSLOduration=2.260879059 podStartE2EDuration="31.134999255s" totalImagesPullingTime="0s" totalInitContainerRuntime="28.874120196s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:09:40.134525841 +0000 UTC m=+1276.446083488" watchObservedRunningTime="2026-07-31 15:09:40.134999255 +0000 UTC m=+1276.446556876"
Jul 31 15:09:41.117886 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:41.117859    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"
Jul 31 15:09:47.127852 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:09:47.127826    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"
Jul 31 15:11:12.221037 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:12.221007    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"
Jul 31 15:11:23.258947 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:23.258898    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd" podUID="ac3b1344-c4e2-4d86-ab8c-3fc7179843c3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.29:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 31 15:11:26.789626 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:26.789585    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"]
Jul 31 15:11:26.790077 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:26.789841    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd" podUID="ac3b1344-c4e2-4d86-ab8c-3fc7179843c3" containerName="kserve-container" containerID="cri-o://81cdbb61613b7266e50ac1fdaa7bc7a9c253578ac86585c82187266a620d36a9" gracePeriod=30
Jul 31 15:11:26.790077 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:26.789874    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd" podUID="ac3b1344-c4e2-4d86-ab8c-3fc7179843c3" containerName="kube-rbac-proxy" containerID="cri-o://96ec58dc852c5dbe5a93294963c6a279b88197667c63705fd679d4a101865a56" gracePeriod=30
Jul 31 15:11:27.122594 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:27.122552    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd" podUID="ac3b1344-c4e2-4d86-ab8c-3fc7179843c3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 31 15:11:27.507148 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:27.507062    2559 generic.go:356] "Generic (PLEG): container finished" podID="ac3b1344-c4e2-4d86-ab8c-3fc7179843c3" containerID="96ec58dc852c5dbe5a93294963c6a279b88197667c63705fd679d4a101865a56" exitCode=2
Jul 31 15:11:27.507274 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:27.507140    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd" event={"ID":"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3","Type":"ContainerDied","Data":"96ec58dc852c5dbe5a93294963c6a279b88197667c63705fd679d4a101865a56"}
Jul 31 15:11:28.243383 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.243359    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"
Jul 31 15:11:28.370596 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.370555    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-cabundle-cert\") pod \"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3\" (UID: \"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3\") "
Jul 31 15:11:28.370791 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.370616    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-kube-rbac-proxy-sar-config\") pod \"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3\" (UID: \"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3\") "
Jul 31 15:11:28.370791 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.370670    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qj7n\" (UniqueName: \"kubernetes.io/projected/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-kube-api-access-8qj7n\") pod \"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3\" (UID: \"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3\") "
Jul 31 15:11:28.370791 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.370693    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-kserve-provision-location\") pod \"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3\" (UID: \"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3\") "
Jul 31 15:11:28.370791 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.370751    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-proxy-tls\") pod \"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3\" (UID: \"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3\") "
Jul 31 15:11:28.371065 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.371045    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ac3b1344-c4e2-4d86-ab8c-3fc7179843c3" (UID: "ac3b1344-c4e2-4d86-ab8c-3fc7179843c3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:11:28.371124 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.371095    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ac3b1344-c4e2-4d86-ab8c-3fc7179843c3" (UID: "ac3b1344-c4e2-4d86-ab8c-3fc7179843c3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:11:28.372924 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.372890    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-kube-api-access-8qj7n" (OuterVolumeSpecName: "kube-api-access-8qj7n") pod "ac3b1344-c4e2-4d86-ab8c-3fc7179843c3" (UID: "ac3b1344-c4e2-4d86-ab8c-3fc7179843c3"). InnerVolumeSpecName "kube-api-access-8qj7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:11:28.373035 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.372947    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ac3b1344-c4e2-4d86-ab8c-3fc7179843c3" (UID: "ac3b1344-c4e2-4d86-ab8c-3fc7179843c3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:11:28.374718 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.374692    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ac3b1344-c4e2-4d86-ab8c-3fc7179843c3" (UID: "ac3b1344-c4e2-4d86-ab8c-3fc7179843c3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:11:28.471506 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.471455    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:11:28.471506 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.471501    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:11:28.471506 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.471513    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qj7n\" (UniqueName: \"kubernetes.io/projected/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-kube-api-access-8qj7n\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:11:28.471506 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.471522    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:11:28.471824 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.471534    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:11:28.512475 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.512447    2559 generic.go:356] "Generic (PLEG): container finished" podID="ac3b1344-c4e2-4d86-ab8c-3fc7179843c3" containerID="81cdbb61613b7266e50ac1fdaa7bc7a9c253578ac86585c82187266a620d36a9" exitCode=0
Jul 31 15:11:28.512616 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.512485    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd" event={"ID":"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3","Type":"ContainerDied","Data":"81cdbb61613b7266e50ac1fdaa7bc7a9c253578ac86585c82187266a620d36a9"}
Jul 31 15:11:28.512616 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.512496    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"
Jul 31 15:11:28.512616 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.512542    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd" event={"ID":"ac3b1344-c4e2-4d86-ab8c-3fc7179843c3","Type":"ContainerDied","Data":"d609f38601e5bd12015ba29309e6823ace735d55e3d80e744062fb658e06f8be"}
Jul 31 15:11:28.512616 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.512569    2559 scope.go:117] "RemoveContainer" containerID="96ec58dc852c5dbe5a93294963c6a279b88197667c63705fd679d4a101865a56"
Jul 31 15:11:28.520943 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.520926    2559 scope.go:117] "RemoveContainer" containerID="81cdbb61613b7266e50ac1fdaa7bc7a9c253578ac86585c82187266a620d36a9"
Jul 31 15:11:28.528238 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.528221    2559 scope.go:117] "RemoveContainer" containerID="bbc8f7c78c3459f24224b8461ff173e3186296dd5e3d7d8a3e22336565ac08f2"
Jul 31 15:11:28.533796 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.533770    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"]
Jul 31 15:11:28.535963 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.535947    2559 scope.go:117] "RemoveContainer" containerID="96ec58dc852c5dbe5a93294963c6a279b88197667c63705fd679d4a101865a56"
Jul 31 15:11:28.536282 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:11:28.536247    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ec58dc852c5dbe5a93294963c6a279b88197667c63705fd679d4a101865a56\": container with ID starting with 96ec58dc852c5dbe5a93294963c6a279b88197667c63705fd679d4a101865a56 not found: ID does not exist" containerID="96ec58dc852c5dbe5a93294963c6a279b88197667c63705fd679d4a101865a56"
Jul 31 15:11:28.536371 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.536292    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ec58dc852c5dbe5a93294963c6a279b88197667c63705fd679d4a101865a56"} err="failed to get container status \"96ec58dc852c5dbe5a93294963c6a279b88197667c63705fd679d4a101865a56\": rpc error: code = NotFound desc = could not find container \"96ec58dc852c5dbe5a93294963c6a279b88197667c63705fd679d4a101865a56\": container with ID starting with 96ec58dc852c5dbe5a93294963c6a279b88197667c63705fd679d4a101865a56 not found: ID does not exist"
Jul 31 15:11:28.536371 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.536318    2559 scope.go:117] "RemoveContainer" containerID="81cdbb61613b7266e50ac1fdaa7bc7a9c253578ac86585c82187266a620d36a9"
Jul 31 15:11:28.536650 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:11:28.536625    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81cdbb61613b7266e50ac1fdaa7bc7a9c253578ac86585c82187266a620d36a9\": container with ID starting with 81cdbb61613b7266e50ac1fdaa7bc7a9c253578ac86585c82187266a620d36a9 not found: ID does not exist" containerID="81cdbb61613b7266e50ac1fdaa7bc7a9c253578ac86585c82187266a620d36a9"
Jul 31 15:11:28.536816 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.536667    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81cdbb61613b7266e50ac1fdaa7bc7a9c253578ac86585c82187266a620d36a9"} err="failed to get container status \"81cdbb61613b7266e50ac1fdaa7bc7a9c253578ac86585c82187266a620d36a9\": rpc error: code = NotFound desc = could not find container \"81cdbb61613b7266e50ac1fdaa7bc7a9c253578ac86585c82187266a620d36a9\": container with ID starting with 81cdbb61613b7266e50ac1fdaa7bc7a9c253578ac86585c82187266a620d36a9 not found: ID does not exist"
Jul 31 15:11:28.536816 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.536690    2559 scope.go:117] "RemoveContainer" containerID="bbc8f7c78c3459f24224b8461ff173e3186296dd5e3d7d8a3e22336565ac08f2"
Jul 31 15:11:28.537026 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:11:28.536988    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbc8f7c78c3459f24224b8461ff173e3186296dd5e3d7d8a3e22336565ac08f2\": container with ID starting with bbc8f7c78c3459f24224b8461ff173e3186296dd5e3d7d8a3e22336565ac08f2 not found: ID does not exist" containerID="bbc8f7c78c3459f24224b8461ff173e3186296dd5e3d7d8a3e22336565ac08f2"
Jul 31 15:11:28.537150 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.537032    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbc8f7c78c3459f24224b8461ff173e3186296dd5e3d7d8a3e22336565ac08f2"} err="failed to get container status \"bbc8f7c78c3459f24224b8461ff173e3186296dd5e3d7d8a3e22336565ac08f2\": rpc error: code = NotFound desc = could not find container \"bbc8f7c78c3459f24224b8461ff173e3186296dd5e3d7d8a3e22336565ac08f2\": container with ID starting with bbc8f7c78c3459f24224b8461ff173e3186296dd5e3d7d8a3e22336565ac08f2 not found: ID does not exist"
Jul 31 15:11:28.538201 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:28.538179    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7v5zd"]
Jul 31 15:11:30.220787 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:30.220722    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac3b1344-c4e2-4d86-ab8c-3fc7179843c3" path="/var/lib/kubelet/pods/ac3b1344-c4e2-4d86-ab8c-3fc7179843c3/volumes"
Jul 31 15:11:31.962010 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:31.961966    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"]
Jul 31 15:11:31.962400 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:31.962248    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac3b1344-c4e2-4d86-ab8c-3fc7179843c3" containerName="kserve-container"
Jul 31 15:11:31.962400 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:31.962259    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac3b1344-c4e2-4d86-ab8c-3fc7179843c3" containerName="kserve-container"
Jul 31 15:11:31.962400 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:31.962273    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac3b1344-c4e2-4d86-ab8c-3fc7179843c3" containerName="storage-initializer"
Jul 31 15:11:31.962400 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:31.962278    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac3b1344-c4e2-4d86-ab8c-3fc7179843c3" containerName="storage-initializer"
Jul 31 15:11:31.962400 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:31.962286    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac3b1344-c4e2-4d86-ab8c-3fc7179843c3" containerName="kube-rbac-proxy"
Jul 31 15:11:31.962400 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:31.962291    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac3b1344-c4e2-4d86-ab8c-3fc7179843c3" containerName="kube-rbac-proxy"
Jul 31 15:11:31.962400 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:31.962341    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac3b1344-c4e2-4d86-ab8c-3fc7179843c3" containerName="kserve-container"
Jul 31 15:11:31.962400 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:31.962351    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac3b1344-c4e2-4d86-ab8c-3fc7179843c3" containerName="kube-rbac-proxy"
Jul 31 15:11:31.965929 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:31.965907    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"
Jul 31 15:11:31.968791 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:31.968765    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 31 15:11:31.968988 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:31.968856    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 15:11:31.968988 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:31.968856    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 15:11:31.968988 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:31.968909    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 15:11:31.968988 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:31.968974    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dd67w\""
Jul 31 15:11:31.969193 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:31.969170    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 31 15:11:31.969714 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:31.969697    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:11:31.982023 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:31.981997    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"]
Jul 31 15:11:32.099509 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:32.099460    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/602aa046-ed4c-403b-b366-73f92941bad2-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9\" (UID: \"602aa046-ed4c-403b-b366-73f92941bad2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"
Jul 31 15:11:32.099509 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:32.099506    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/602aa046-ed4c-403b-b366-73f92941bad2-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9\" (UID: \"602aa046-ed4c-403b-b366-73f92941bad2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"
Jul 31 15:11:32.099801 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:32.099545    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/602aa046-ed4c-403b-b366-73f92941bad2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9\" (UID: \"602aa046-ed4c-403b-b366-73f92941bad2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"
Jul 31 15:11:32.099801 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:32.099624    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/602aa046-ed4c-403b-b366-73f92941bad2-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9\" (UID: \"602aa046-ed4c-403b-b366-73f92941bad2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"
Jul 31 15:11:32.099801 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:32.099676    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n66x6\" (UniqueName: \"kubernetes.io/projected/602aa046-ed4c-403b-b366-73f92941bad2-kube-api-access-n66x6\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9\" (UID: \"602aa046-ed4c-403b-b366-73f92941bad2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"
Jul 31 15:11:32.201106 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:32.201061    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n66x6\" (UniqueName: \"kubernetes.io/projected/602aa046-ed4c-403b-b366-73f92941bad2-kube-api-access-n66x6\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9\" (UID: \"602aa046-ed4c-403b-b366-73f92941bad2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"
Jul 31 15:11:32.201106 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:32.201115    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/602aa046-ed4c-403b-b366-73f92941bad2-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9\" (UID: \"602aa046-ed4c-403b-b366-73f92941bad2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"
Jul 31 15:11:32.201324 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:32.201235    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/602aa046-ed4c-403b-b366-73f92941bad2-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9\" (UID: \"602aa046-ed4c-403b-b366-73f92941bad2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"
Jul 31 15:11:32.201324 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:32.201304    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/602aa046-ed4c-403b-b366-73f92941bad2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9\" (UID: \"602aa046-ed4c-403b-b366-73f92941bad2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"
Jul 31 15:11:32.201392 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:32.201346    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/602aa046-ed4c-403b-b366-73f92941bad2-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9\" (UID: \"602aa046-ed4c-403b-b366-73f92941bad2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"
Jul 31 15:11:32.201697 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:32.201672    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/602aa046-ed4c-403b-b366-73f92941bad2-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9\" (UID: \"602aa046-ed4c-403b-b366-73f92941bad2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"
Jul 31 15:11:32.201841 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:32.201825    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/602aa046-ed4c-403b-b366-73f92941bad2-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9\" (UID: \"602aa046-ed4c-403b-b366-73f92941bad2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"
Jul 31 15:11:32.201913 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:32.201897    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/602aa046-ed4c-403b-b366-73f92941bad2-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9\" (UID: \"602aa046-ed4c-403b-b366-73f92941bad2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"
Jul 31 15:11:32.203861 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:32.203843    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/602aa046-ed4c-403b-b366-73f92941bad2-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9\" (UID: \"602aa046-ed4c-403b-b366-73f92941bad2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"
Jul 31 15:11:32.210434 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:32.210404    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n66x6\" (UniqueName: \"kubernetes.io/projected/602aa046-ed4c-403b-b366-73f92941bad2-kube-api-access-n66x6\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9\" (UID: \"602aa046-ed4c-403b-b366-73f92941bad2\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"
Jul 31 15:11:32.276718 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:32.276630    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"
Jul 31 15:11:32.421041 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:32.421001    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"]
Jul 31 15:11:32.456703 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:11:32.456676    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod602aa046_ed4c_403b_b366_73f92941bad2.slice/crio-ba8f58adbf145678fbe733212b2c28839509b0cdeb138e1b688063fd7b0832ea WatchSource:0}: Error finding container ba8f58adbf145678fbe733212b2c28839509b0cdeb138e1b688063fd7b0832ea: Status 404 returned error can't find the container with id ba8f58adbf145678fbe733212b2c28839509b0cdeb138e1b688063fd7b0832ea
Jul 31 15:11:32.532553 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:32.532468    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9" event={"ID":"602aa046-ed4c-403b-b366-73f92941bad2","Type":"ContainerStarted","Data":"4294c2aa2e7db331a97d4c6eea07c59f91b83f86e4ce9c8be41e1b77d985b88d"}
Jul 31 15:11:32.532553 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:32.532533    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9" event={"ID":"602aa046-ed4c-403b-b366-73f92941bad2","Type":"ContainerStarted","Data":"ba8f58adbf145678fbe733212b2c28839509b0cdeb138e1b688063fd7b0832ea"}
Jul 31 15:11:53.609322 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:53.609289    2559 generic.go:356] "Generic (PLEG): container finished" podID="602aa046-ed4c-403b-b366-73f92941bad2" containerID="4294c2aa2e7db331a97d4c6eea07c59f91b83f86e4ce9c8be41e1b77d985b88d" exitCode=0
Jul 31 15:11:53.609754 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:53.609365    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9" event={"ID":"602aa046-ed4c-403b-b366-73f92941bad2","Type":"ContainerDied","Data":"4294c2aa2e7db331a97d4c6eea07c59f91b83f86e4ce9c8be41e1b77d985b88d"}
Jul 31 15:11:54.614573 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:54.614536    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9" event={"ID":"602aa046-ed4c-403b-b366-73f92941bad2","Type":"ContainerStarted","Data":"f528728ec5e898bc07b82d24731a112b35ed6586b18449b21500496325f6478e"}
Jul 31 15:11:54.615105 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:54.614639    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9" event={"ID":"602aa046-ed4c-403b-b366-73f92941bad2","Type":"ContainerStarted","Data":"6a550e21fc9adc77e429fa35f6352c2ef1452da27cf07668f273ea95fd04ded5"}
Jul 31 15:11:54.615105 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:54.614871    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"
Jul 31 15:11:54.636666 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:54.636611    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9" podStartSLOduration=3.019899185 podStartE2EDuration="23.63659417s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.616694985s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:11:54.634923215 +0000 UTC m=+1410.946480888" watchObservedRunningTime="2026-07-31 15:11:54.63659417 +0000 UTC m=+1410.948151792"
Jul 31 15:11:55.618073 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:11:55.618038    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"
Jul 31 15:12:01.629203 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:12:01.629176    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"
Jul 31 15:13:24.432048 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:24.432018    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"]
Jul 31 15:13:24.435304 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:24.435287    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"
Jul 31 15:13:24.437912 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:24.437891    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Jul 31 15:13:24.438093 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:24.438075    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Jul 31 15:13:24.447013 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:24.446991    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"]
Jul 31 15:13:24.535786 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:24.535757    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/115552c7-ff25-4bee-9a28-5a0093a60336-proxy-tls\") pod \"isvc-sklearn-predictor-5f8888fcd7-r8qbz\" (UID: \"115552c7-ff25-4bee-9a28-5a0093a60336\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"
Jul 31 15:13:24.535786 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:24.535788    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/115552c7-ff25-4bee-9a28-5a0093a60336-cabundle-cert\") pod \"isvc-sklearn-predictor-5f8888fcd7-r8qbz\" (UID: \"115552c7-ff25-4bee-9a28-5a0093a60336\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"
Jul 31 15:13:24.535993 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:24.535818    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r94zs\" (UniqueName: \"kubernetes.io/projected/115552c7-ff25-4bee-9a28-5a0093a60336-kube-api-access-r94zs\") pod \"isvc-sklearn-predictor-5f8888fcd7-r8qbz\" (UID: \"115552c7-ff25-4bee-9a28-5a0093a60336\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"
Jul 31 15:13:24.535993 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:24.535895    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/115552c7-ff25-4bee-9a28-5a0093a60336-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5f8888fcd7-r8qbz\" (UID: \"115552c7-ff25-4bee-9a28-5a0093a60336\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"
Jul 31 15:13:24.535993 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:24.535920    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/115552c7-ff25-4bee-9a28-5a0093a60336-kserve-provision-location\") pod \"isvc-sklearn-predictor-5f8888fcd7-r8qbz\" (UID: \"115552c7-ff25-4bee-9a28-5a0093a60336\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"
Jul 31 15:13:24.636567 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:24.636528    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/115552c7-ff25-4bee-9a28-5a0093a60336-proxy-tls\") pod \"isvc-sklearn-predictor-5f8888fcd7-r8qbz\" (UID: \"115552c7-ff25-4bee-9a28-5a0093a60336\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"
Jul 31 15:13:24.636567 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:24.636566    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/115552c7-ff25-4bee-9a28-5a0093a60336-cabundle-cert\") pod \"isvc-sklearn-predictor-5f8888fcd7-r8qbz\" (UID: \"115552c7-ff25-4bee-9a28-5a0093a60336\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"
Jul 31 15:13:24.636842 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:24.636597    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r94zs\" (UniqueName: \"kubernetes.io/projected/115552c7-ff25-4bee-9a28-5a0093a60336-kube-api-access-r94zs\") pod \"isvc-sklearn-predictor-5f8888fcd7-r8qbz\" (UID: \"115552c7-ff25-4bee-9a28-5a0093a60336\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"
Jul 31 15:13:24.636842 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:24.636631    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/115552c7-ff25-4bee-9a28-5a0093a60336-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5f8888fcd7-r8qbz\" (UID: \"115552c7-ff25-4bee-9a28-5a0093a60336\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"
Jul 31 15:13:24.636842 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:24.636647    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/115552c7-ff25-4bee-9a28-5a0093a60336-kserve-provision-location\") pod \"isvc-sklearn-predictor-5f8888fcd7-r8qbz\" (UID: \"115552c7-ff25-4bee-9a28-5a0093a60336\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"
Jul 31 15:13:24.637136 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:24.637114    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/115552c7-ff25-4bee-9a28-5a0093a60336-kserve-provision-location\") pod \"isvc-sklearn-predictor-5f8888fcd7-r8qbz\" (UID: \"115552c7-ff25-4bee-9a28-5a0093a60336\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"
Jul 31 15:13:24.637446 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:24.637397    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/115552c7-ff25-4bee-9a28-5a0093a60336-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5f8888fcd7-r8qbz\" (UID: \"115552c7-ff25-4bee-9a28-5a0093a60336\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"
Jul 31 15:13:24.637446 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:24.637400    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/115552c7-ff25-4bee-9a28-5a0093a60336-cabundle-cert\") pod \"isvc-sklearn-predictor-5f8888fcd7-r8qbz\" (UID: \"115552c7-ff25-4bee-9a28-5a0093a60336\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"
Jul 31 15:13:24.639136 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:24.639115    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/115552c7-ff25-4bee-9a28-5a0093a60336-proxy-tls\") pod \"isvc-sklearn-predictor-5f8888fcd7-r8qbz\" (UID: \"115552c7-ff25-4bee-9a28-5a0093a60336\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"
Jul 31 15:13:24.645228 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:24.645202    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r94zs\" (UniqueName: \"kubernetes.io/projected/115552c7-ff25-4bee-9a28-5a0093a60336-kube-api-access-r94zs\") pod \"isvc-sklearn-predictor-5f8888fcd7-r8qbz\" (UID: \"115552c7-ff25-4bee-9a28-5a0093a60336\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"
Jul 31 15:13:24.746525 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:24.746435    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"
Jul 31 15:13:24.880101 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:24.880076    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"]
Jul 31 15:13:24.882358 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:13:24.882330    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod115552c7_ff25_4bee_9a28_5a0093a60336.slice/crio-e0bac958b675a9d67841395e8e67a040d6dcf985707f092e8815eed2e51fb8b9 WatchSource:0}: Error finding container e0bac958b675a9d67841395e8e67a040d6dcf985707f092e8815eed2e51fb8b9: Status 404 returned error can't find the container with id e0bac958b675a9d67841395e8e67a040d6dcf985707f092e8815eed2e51fb8b9
Jul 31 15:13:24.942274 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:24.942245    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz" event={"ID":"115552c7-ff25-4bee-9a28-5a0093a60336","Type":"ContainerStarted","Data":"d6c2355a789a65cacbeb671a59246a6d4d5d67b42ecacedcfad4e107822cdd8d"}
Jul 31 15:13:24.942371 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:24.942306    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz" event={"ID":"115552c7-ff25-4bee-9a28-5a0093a60336","Type":"ContainerStarted","Data":"e0bac958b675a9d67841395e8e67a040d6dcf985707f092e8815eed2e51fb8b9"}
Jul 31 15:13:28.957048 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:28.957020    2559 generic.go:356] "Generic (PLEG): container finished" podID="115552c7-ff25-4bee-9a28-5a0093a60336" containerID="d6c2355a789a65cacbeb671a59246a6d4d5d67b42ecacedcfad4e107822cdd8d" exitCode=0
Jul 31 15:13:28.957397 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:28.957078    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz" event={"ID":"115552c7-ff25-4bee-9a28-5a0093a60336","Type":"ContainerDied","Data":"d6c2355a789a65cacbeb671a59246a6d4d5d67b42ecacedcfad4e107822cdd8d"}
Jul 31 15:13:29.962439 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:29.962401    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz" event={"ID":"115552c7-ff25-4bee-9a28-5a0093a60336","Type":"ContainerStarted","Data":"7896c5e5a5b30bf2593deb1e5d65aec6d3e035e983ac193a54c0404385a44097"}
Jul 31 15:13:29.962910 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:29.962468    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz" event={"ID":"115552c7-ff25-4bee-9a28-5a0093a60336","Type":"ContainerStarted","Data":"86bfe11760ccdfd20bfd54c35ac4c9e9676a80e10504e791a1d39a03a7a632b9"}
Jul 31 15:13:29.962910 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:29.962662    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"
Jul 31 15:13:29.983252 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:29.983202    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz" podStartSLOduration=2.021632706 podStartE2EDuration="5.983186386s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.96155368s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:13:29.981460822 +0000 UTC m=+1506.293018439" watchObservedRunningTime="2026-07-31 15:13:29.983186386 +0000 UTC m=+1506.294744075"
Jul 31 15:13:30.966202 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:30.966165    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"
Jul 31 15:13:30.967601 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:30.967572    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz" podUID="115552c7-ff25-4bee-9a28-5a0093a60336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 31 15:13:31.220606 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:31.220533    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"
Jul 31 15:13:31.970031 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:31.969981    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz" podUID="115552c7-ff25-4bee-9a28-5a0093a60336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 31 15:13:36.974647 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:36.974618    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"
Jul 31 15:13:36.975163 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:36.975134    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz" podUID="115552c7-ff25-4bee-9a28-5a0093a60336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 31 15:13:42.256936 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:42.256891    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9" podUID="602aa046-ed4c-403b-b366-73f92941bad2" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.30:8080/v2/models/isvc-autogluon-v2-noslash/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 31 15:13:42.574054 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:42.573972    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"]
Jul 31 15:13:42.574241 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:42.574221    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9" podUID="602aa046-ed4c-403b-b366-73f92941bad2" containerName="kserve-container" containerID="cri-o://6a550e21fc9adc77e429fa35f6352c2ef1452da27cf07668f273ea95fd04ded5" gracePeriod=30
Jul 31 15:13:42.574339 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:42.574313    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9" podUID="602aa046-ed4c-403b-b366-73f92941bad2" containerName="kube-rbac-proxy" containerID="cri-o://f528728ec5e898bc07b82d24731a112b35ed6586b18449b21500496325f6478e" gracePeriod=30
Jul 31 15:13:43.014064 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:43.014032    2559 generic.go:356] "Generic (PLEG): container finished" podID="602aa046-ed4c-403b-b366-73f92941bad2" containerID="f528728ec5e898bc07b82d24731a112b35ed6586b18449b21500496325f6478e" exitCode=2
Jul 31 15:13:43.014228 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:43.014100    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9" event={"ID":"602aa046-ed4c-403b-b366-73f92941bad2","Type":"ContainerDied","Data":"f528728ec5e898bc07b82d24731a112b35ed6586b18449b21500496325f6478e"}
Jul 31 15:13:44.013828 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.013801    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"
Jul 31 15:13:44.019452 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.019428    2559 generic.go:356] "Generic (PLEG): container finished" podID="602aa046-ed4c-403b-b366-73f92941bad2" containerID="6a550e21fc9adc77e429fa35f6352c2ef1452da27cf07668f273ea95fd04ded5" exitCode=0
Jul 31 15:13:44.019563 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.019484    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"
Jul 31 15:13:44.019563 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.019501    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9" event={"ID":"602aa046-ed4c-403b-b366-73f92941bad2","Type":"ContainerDied","Data":"6a550e21fc9adc77e429fa35f6352c2ef1452da27cf07668f273ea95fd04ded5"}
Jul 31 15:13:44.019563 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.019556    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9" event={"ID":"602aa046-ed4c-403b-b366-73f92941bad2","Type":"ContainerDied","Data":"ba8f58adbf145678fbe733212b2c28839509b0cdeb138e1b688063fd7b0832ea"}
Jul 31 15:13:44.019679 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.019582    2559 scope.go:117] "RemoveContainer" containerID="f528728ec5e898bc07b82d24731a112b35ed6586b18449b21500496325f6478e"
Jul 31 15:13:44.027243 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.027227    2559 scope.go:117] "RemoveContainer" containerID="6a550e21fc9adc77e429fa35f6352c2ef1452da27cf07668f273ea95fd04ded5"
Jul 31 15:13:44.034454 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.034432    2559 scope.go:117] "RemoveContainer" containerID="4294c2aa2e7db331a97d4c6eea07c59f91b83f86e4ce9c8be41e1b77d985b88d"
Jul 31 15:13:44.043404 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.043384    2559 scope.go:117] "RemoveContainer" containerID="f528728ec5e898bc07b82d24731a112b35ed6586b18449b21500496325f6478e"
Jul 31 15:13:44.043671 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:13:44.043650    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f528728ec5e898bc07b82d24731a112b35ed6586b18449b21500496325f6478e\": container with ID starting with f528728ec5e898bc07b82d24731a112b35ed6586b18449b21500496325f6478e not found: ID does not exist" containerID="f528728ec5e898bc07b82d24731a112b35ed6586b18449b21500496325f6478e"
Jul 31 15:13:44.043820 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.043678    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f528728ec5e898bc07b82d24731a112b35ed6586b18449b21500496325f6478e"} err="failed to get container status \"f528728ec5e898bc07b82d24731a112b35ed6586b18449b21500496325f6478e\": rpc error: code = NotFound desc = could not find container \"f528728ec5e898bc07b82d24731a112b35ed6586b18449b21500496325f6478e\": container with ID starting with f528728ec5e898bc07b82d24731a112b35ed6586b18449b21500496325f6478e not found: ID does not exist"
Jul 31 15:13:44.043820 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.043696    2559 scope.go:117] "RemoveContainer" containerID="6a550e21fc9adc77e429fa35f6352c2ef1452da27cf07668f273ea95fd04ded5"
Jul 31 15:13:44.044012 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:13:44.043996    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a550e21fc9adc77e429fa35f6352c2ef1452da27cf07668f273ea95fd04ded5\": container with ID starting with 6a550e21fc9adc77e429fa35f6352c2ef1452da27cf07668f273ea95fd04ded5 not found: ID does not exist" containerID="6a550e21fc9adc77e429fa35f6352c2ef1452da27cf07668f273ea95fd04ded5"
Jul 31 15:13:44.044053 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.044018    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a550e21fc9adc77e429fa35f6352c2ef1452da27cf07668f273ea95fd04ded5"} err="failed to get container status \"6a550e21fc9adc77e429fa35f6352c2ef1452da27cf07668f273ea95fd04ded5\": rpc error: code = NotFound desc = could not find container \"6a550e21fc9adc77e429fa35f6352c2ef1452da27cf07668f273ea95fd04ded5\": container with ID starting with 6a550e21fc9adc77e429fa35f6352c2ef1452da27cf07668f273ea95fd04ded5 not found: ID does not exist"
Jul 31 15:13:44.044053 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.044037    2559 scope.go:117] "RemoveContainer" containerID="4294c2aa2e7db331a97d4c6eea07c59f91b83f86e4ce9c8be41e1b77d985b88d"
Jul 31 15:13:44.044304 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:13:44.044286    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4294c2aa2e7db331a97d4c6eea07c59f91b83f86e4ce9c8be41e1b77d985b88d\": container with ID starting with 4294c2aa2e7db331a97d4c6eea07c59f91b83f86e4ce9c8be41e1b77d985b88d not found: ID does not exist" containerID="4294c2aa2e7db331a97d4c6eea07c59f91b83f86e4ce9c8be41e1b77d985b88d"
Jul 31 15:13:44.044347 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.044309    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4294c2aa2e7db331a97d4c6eea07c59f91b83f86e4ce9c8be41e1b77d985b88d"} err="failed to get container status \"4294c2aa2e7db331a97d4c6eea07c59f91b83f86e4ce9c8be41e1b77d985b88d\": rpc error: code = NotFound desc = could not find container \"4294c2aa2e7db331a97d4c6eea07c59f91b83f86e4ce9c8be41e1b77d985b88d\": container with ID starting with 4294c2aa2e7db331a97d4c6eea07c59f91b83f86e4ce9c8be41e1b77d985b88d not found: ID does not exist"
Jul 31 15:13:44.089938 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.089848    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/602aa046-ed4c-403b-b366-73f92941bad2-kube-rbac-proxy-sar-config\") pod \"602aa046-ed4c-403b-b366-73f92941bad2\" (UID: \"602aa046-ed4c-403b-b366-73f92941bad2\") "
Jul 31 15:13:44.089938 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.089905    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n66x6\" (UniqueName: \"kubernetes.io/projected/602aa046-ed4c-403b-b366-73f92941bad2-kube-api-access-n66x6\") pod \"602aa046-ed4c-403b-b366-73f92941bad2\" (UID: \"602aa046-ed4c-403b-b366-73f92941bad2\") "
Jul 31 15:13:44.090152 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.089944    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/602aa046-ed4c-403b-b366-73f92941bad2-cabundle-cert\") pod \"602aa046-ed4c-403b-b366-73f92941bad2\" (UID: \"602aa046-ed4c-403b-b366-73f92941bad2\") "
Jul 31 15:13:44.090152 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.089970    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/602aa046-ed4c-403b-b366-73f92941bad2-kserve-provision-location\") pod \"602aa046-ed4c-403b-b366-73f92941bad2\" (UID: \"602aa046-ed4c-403b-b366-73f92941bad2\") "
Jul 31 15:13:44.090152 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.089993    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/602aa046-ed4c-403b-b366-73f92941bad2-proxy-tls\") pod \"602aa046-ed4c-403b-b366-73f92941bad2\" (UID: \"602aa046-ed4c-403b-b366-73f92941bad2\") "
Jul 31 15:13:44.090301 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.090270    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/602aa046-ed4c-403b-b366-73f92941bad2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "602aa046-ed4c-403b-b366-73f92941bad2" (UID: "602aa046-ed4c-403b-b366-73f92941bad2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:13:44.090345 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.090300    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/602aa046-ed4c-403b-b366-73f92941bad2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "602aa046-ed4c-403b-b366-73f92941bad2" (UID: "602aa046-ed4c-403b-b366-73f92941bad2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:13:44.092183 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.092157    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/602aa046-ed4c-403b-b366-73f92941bad2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "602aa046-ed4c-403b-b366-73f92941bad2" (UID: "602aa046-ed4c-403b-b366-73f92941bad2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:13:44.092342 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.092324    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/602aa046-ed4c-403b-b366-73f92941bad2-kube-api-access-n66x6" (OuterVolumeSpecName: "kube-api-access-n66x6") pod "602aa046-ed4c-403b-b366-73f92941bad2" (UID: "602aa046-ed4c-403b-b366-73f92941bad2"). InnerVolumeSpecName "kube-api-access-n66x6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:13:44.095984 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.095960    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/602aa046-ed4c-403b-b366-73f92941bad2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "602aa046-ed4c-403b-b366-73f92941bad2" (UID: "602aa046-ed4c-403b-b366-73f92941bad2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:13:44.191030 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.190995    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/602aa046-ed4c-403b-b366-73f92941bad2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:13:44.191030 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.191023    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n66x6\" (UniqueName: \"kubernetes.io/projected/602aa046-ed4c-403b-b366-73f92941bad2-kube-api-access-n66x6\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:13:44.191030 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.191034    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/602aa046-ed4c-403b-b366-73f92941bad2-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:13:44.191030 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.191042    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/602aa046-ed4c-403b-b366-73f92941bad2-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:13:44.191299 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.191051    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/602aa046-ed4c-403b-b366-73f92941bad2-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:13:44.336280 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.336246    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"]
Jul 31 15:13:44.341351 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:44.341290    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-f5tb9"]
Jul 31 15:13:46.220929 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:46.220826    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="602aa046-ed4c-403b-b366-73f92941bad2" path="/var/lib/kubelet/pods/602aa046-ed4c-403b-b366-73f92941bad2/volumes"
Jul 31 15:13:46.976141 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:46.976099    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz" podUID="115552c7-ff25-4bee-9a28-5a0093a60336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 31 15:13:47.574695 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:47.574658    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"]
Jul 31 15:13:47.575086 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:47.574966    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="602aa046-ed4c-403b-b366-73f92941bad2" containerName="storage-initializer"
Jul 31 15:13:47.575086 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:47.574977    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="602aa046-ed4c-403b-b366-73f92941bad2" containerName="storage-initializer"
Jul 31 15:13:47.575086 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:47.574990    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="602aa046-ed4c-403b-b366-73f92941bad2" containerName="kserve-container"
Jul 31 15:13:47.575086 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:47.574998    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="602aa046-ed4c-403b-b366-73f92941bad2" containerName="kserve-container"
Jul 31 15:13:47.575086 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:47.575010    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="602aa046-ed4c-403b-b366-73f92941bad2" containerName="kube-rbac-proxy"
Jul 31 15:13:47.575086 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:47.575016    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="602aa046-ed4c-403b-b366-73f92941bad2" containerName="kube-rbac-proxy"
Jul 31 15:13:47.575086 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:47.575067    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="602aa046-ed4c-403b-b366-73f92941bad2" containerName="kserve-container"
Jul 31 15:13:47.575086 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:47.575080    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="602aa046-ed4c-403b-b366-73f92941bad2" containerName="kube-rbac-proxy"
Jul 31 15:13:47.579932 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:47.579904    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"
Jul 31 15:13:47.582455 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:47.582434    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 31 15:13:47.582858 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:47.582837    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 31 15:13:47.589072 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:47.589049    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"]
Jul 31 15:13:47.617357 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:47.617326    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4f93428-2137-428f-ad84-42e9db3e1440-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8\" (UID: \"e4f93428-2137-428f-ad84-42e9db3e1440\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"
Jul 31 15:13:47.617357 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:47.617367    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hscnm\" (UniqueName: \"kubernetes.io/projected/e4f93428-2137-428f-ad84-42e9db3e1440-kube-api-access-hscnm\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8\" (UID: \"e4f93428-2137-428f-ad84-42e9db3e1440\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"
Jul 31 15:13:47.617627 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:47.617391    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4f93428-2137-428f-ad84-42e9db3e1440-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8\" (UID: \"e4f93428-2137-428f-ad84-42e9db3e1440\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"
Jul 31 15:13:47.617627 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:47.617468    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4f93428-2137-428f-ad84-42e9db3e1440-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8\" (UID: \"e4f93428-2137-428f-ad84-42e9db3e1440\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"
Jul 31 15:13:47.617627 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:47.617544    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4f93428-2137-428f-ad84-42e9db3e1440-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8\" (UID: \"e4f93428-2137-428f-ad84-42e9db3e1440\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"
Jul 31 15:13:47.718673 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:47.718636    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4f93428-2137-428f-ad84-42e9db3e1440-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8\" (UID: \"e4f93428-2137-428f-ad84-42e9db3e1440\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"
Jul 31 15:13:47.718886 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:47.718685    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hscnm\" (UniqueName: \"kubernetes.io/projected/e4f93428-2137-428f-ad84-42e9db3e1440-kube-api-access-hscnm\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8\" (UID: \"e4f93428-2137-428f-ad84-42e9db3e1440\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"
Jul 31 15:13:47.718886 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:47.718715    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4f93428-2137-428f-ad84-42e9db3e1440-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8\" (UID: \"e4f93428-2137-428f-ad84-42e9db3e1440\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"
Jul 31 15:13:47.718886 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:47.718770    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4f93428-2137-428f-ad84-42e9db3e1440-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8\" (UID: \"e4f93428-2137-428f-ad84-42e9db3e1440\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"
Jul 31 15:13:47.718886 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:47.718822    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4f93428-2137-428f-ad84-42e9db3e1440-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8\" (UID: \"e4f93428-2137-428f-ad84-42e9db3e1440\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"
Jul 31 15:13:47.718886 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:13:47.718878    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-serving-cert: secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Jul 31 15:13:47.719140 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:13:47.718944    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4f93428-2137-428f-ad84-42e9db3e1440-proxy-tls podName:e4f93428-2137-428f-ad84-42e9db3e1440 nodeName:}" failed. No retries permitted until 2026-07-31 15:13:48.218926663 +0000 UTC m=+1524.530484264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e4f93428-2137-428f-ad84-42e9db3e1440-proxy-tls") pod "isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8" (UID: "e4f93428-2137-428f-ad84-42e9db3e1440") : secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Jul 31 15:13:47.719254 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:47.719236    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4f93428-2137-428f-ad84-42e9db3e1440-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8\" (UID: \"e4f93428-2137-428f-ad84-42e9db3e1440\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"
Jul 31 15:13:47.719528 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:47.719507    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4f93428-2137-428f-ad84-42e9db3e1440-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8\" (UID: \"e4f93428-2137-428f-ad84-42e9db3e1440\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"
Jul 31 15:13:47.719608 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:47.719506    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4f93428-2137-428f-ad84-42e9db3e1440-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8\" (UID: \"e4f93428-2137-428f-ad84-42e9db3e1440\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"
Jul 31 15:13:47.726819 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:47.726797    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hscnm\" (UniqueName: \"kubernetes.io/projected/e4f93428-2137-428f-ad84-42e9db3e1440-kube-api-access-hscnm\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8\" (UID: \"e4f93428-2137-428f-ad84-42e9db3e1440\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"
Jul 31 15:13:48.223506 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:48.223478    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4f93428-2137-428f-ad84-42e9db3e1440-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8\" (UID: \"e4f93428-2137-428f-ad84-42e9db3e1440\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"
Jul 31 15:13:48.225991 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:48.225972    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4f93428-2137-428f-ad84-42e9db3e1440-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8\" (UID: \"e4f93428-2137-428f-ad84-42e9db3e1440\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"
Jul 31 15:13:48.491092 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:48.491004    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"
Jul 31 15:13:48.627593 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:48.627551    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"]
Jul 31 15:13:48.630446 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:13:48.630417    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f93428_2137_428f_ad84_42e9db3e1440.slice/crio-9fd17ae8630ac453679283459d1f8e2a52c0571ace9e885fc477760cca54fa82 WatchSource:0}: Error finding container 9fd17ae8630ac453679283459d1f8e2a52c0571ace9e885fc477760cca54fa82: Status 404 returned error can't find the container with id 9fd17ae8630ac453679283459d1f8e2a52c0571ace9e885fc477760cca54fa82
Jul 31 15:13:49.041297 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:49.041234    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8" event={"ID":"e4f93428-2137-428f-ad84-42e9db3e1440","Type":"ContainerStarted","Data":"707b94e084414f5c9d486e1de8dda3dc9d666db542e5010aa14e17cef04da7d0"}
Jul 31 15:13:49.041460 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:49.041378    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8" event={"ID":"e4f93428-2137-428f-ad84-42e9db3e1440","Type":"ContainerStarted","Data":"9fd17ae8630ac453679283459d1f8e2a52c0571ace9e885fc477760cca54fa82"}
Jul 31 15:13:56.975592 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:56.975550    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz" podUID="115552c7-ff25-4bee-9a28-5a0093a60336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 31 15:13:59.080349 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:59.080318    2559 generic.go:356] "Generic (PLEG): container finished" podID="e4f93428-2137-428f-ad84-42e9db3e1440" containerID="707b94e084414f5c9d486e1de8dda3dc9d666db542e5010aa14e17cef04da7d0" exitCode=0
Jul 31 15:13:59.080851 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:13:59.080394    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8" event={"ID":"e4f93428-2137-428f-ad84-42e9db3e1440","Type":"ContainerDied","Data":"707b94e084414f5c9d486e1de8dda3dc9d666db542e5010aa14e17cef04da7d0"}
Jul 31 15:14:00.086725 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:00.086682    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8" event={"ID":"e4f93428-2137-428f-ad84-42e9db3e1440","Type":"ContainerStarted","Data":"d50ac78236172a780dc8b8e255280c90418c1f39344c2c3f3f03d2a755a0dcd5"}
Jul 31 15:14:00.087177 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:00.086765    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8" event={"ID":"e4f93428-2137-428f-ad84-42e9db3e1440","Type":"ContainerStarted","Data":"8aa83e1c49e786eeac6afff962d5d3f3ca49f53e512e3f1b2e7c7316a3af06f8"}
Jul 31 15:14:00.087177 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:00.086947    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"
Jul 31 15:14:00.107314 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:00.107268    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8" podStartSLOduration=3.548701824 podStartE2EDuration="13.1072533s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.558551476s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:14:00.105971493 +0000 UTC m=+1536.417529116" watchObservedRunningTime="2026-07-31 15:14:00.1072533 +0000 UTC m=+1536.418810921"
Jul 31 15:14:01.090972 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:01.090942    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"
Jul 31 15:14:01.092367 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:01.092339    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8" podUID="e4f93428-2137-428f-ad84-42e9db3e1440" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 31 15:14:02.094479 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:02.094434    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8" podUID="e4f93428-2137-428f-ad84-42e9db3e1440" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 31 15:14:06.975127 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:06.975087    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz" podUID="115552c7-ff25-4bee-9a28-5a0093a60336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 31 15:14:07.099329 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:07.099303    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"
Jul 31 15:14:07.099884 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:07.099853    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8" podUID="e4f93428-2137-428f-ad84-42e9db3e1440" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 31 15:14:16.975517 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:16.975469    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz" podUID="115552c7-ff25-4bee-9a28-5a0093a60336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 31 15:14:17.100462 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:17.100433    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"
Jul 31 15:14:26.975180 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:26.975134    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz" podUID="115552c7-ff25-4bee-9a28-5a0093a60336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 31 15:14:30.014988 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:30.014948    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"]
Jul 31 15:14:30.015404 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:30.015187    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8" podUID="e4f93428-2137-428f-ad84-42e9db3e1440" containerName="kserve-container" containerID="cri-o://8aa83e1c49e786eeac6afff962d5d3f3ca49f53e512e3f1b2e7c7316a3af06f8" gracePeriod=30
Jul 31 15:14:30.015404 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:30.015266    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8" podUID="e4f93428-2137-428f-ad84-42e9db3e1440" containerName="kube-rbac-proxy" containerID="cri-o://d50ac78236172a780dc8b8e255280c90418c1f39344c2c3f3f03d2a755a0dcd5" gracePeriod=30
Jul 31 15:14:30.206234 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:30.206203    2559 generic.go:356] "Generic (PLEG): container finished" podID="e4f93428-2137-428f-ad84-42e9db3e1440" containerID="d50ac78236172a780dc8b8e255280c90418c1f39344c2c3f3f03d2a755a0dcd5" exitCode=2
Jul 31 15:14:30.206416 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:30.206289    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8" event={"ID":"e4f93428-2137-428f-ad84-42e9db3e1440","Type":"ContainerDied","Data":"d50ac78236172a780dc8b8e255280c90418c1f39344c2c3f3f03d2a755a0dcd5"}
Jul 31 15:14:31.660497 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:31.660467    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"
Jul 31 15:14:31.673395 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:31.673369    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4f93428-2137-428f-ad84-42e9db3e1440-proxy-tls\") pod \"e4f93428-2137-428f-ad84-42e9db3e1440\" (UID: \"e4f93428-2137-428f-ad84-42e9db3e1440\") "
Jul 31 15:14:31.673565 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:31.673423    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hscnm\" (UniqueName: \"kubernetes.io/projected/e4f93428-2137-428f-ad84-42e9db3e1440-kube-api-access-hscnm\") pod \"e4f93428-2137-428f-ad84-42e9db3e1440\" (UID: \"e4f93428-2137-428f-ad84-42e9db3e1440\") "
Jul 31 15:14:31.673565 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:31.673461    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4f93428-2137-428f-ad84-42e9db3e1440-cabundle-cert\") pod \"e4f93428-2137-428f-ad84-42e9db3e1440\" (UID: \"e4f93428-2137-428f-ad84-42e9db3e1440\") "
Jul 31 15:14:31.673565 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:31.673502    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4f93428-2137-428f-ad84-42e9db3e1440-kube-rbac-proxy-sar-config\") pod \"e4f93428-2137-428f-ad84-42e9db3e1440\" (UID: \"e4f93428-2137-428f-ad84-42e9db3e1440\") "
Jul 31 15:14:31.673565 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:31.673544    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4f93428-2137-428f-ad84-42e9db3e1440-kserve-provision-location\") pod \"e4f93428-2137-428f-ad84-42e9db3e1440\" (UID: \"e4f93428-2137-428f-ad84-42e9db3e1440\") "
Jul 31 15:14:31.673941 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:31.673913    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f93428-2137-428f-ad84-42e9db3e1440-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e4f93428-2137-428f-ad84-42e9db3e1440" (UID: "e4f93428-2137-428f-ad84-42e9db3e1440"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:14:31.674008 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:31.673917    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f93428-2137-428f-ad84-42e9db3e1440-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e4f93428-2137-428f-ad84-42e9db3e1440" (UID: "e4f93428-2137-428f-ad84-42e9db3e1440"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:14:31.674540 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:31.674514    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4f93428-2137-428f-ad84-42e9db3e1440-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e4f93428-2137-428f-ad84-42e9db3e1440" (UID: "e4f93428-2137-428f-ad84-42e9db3e1440"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:14:31.675718 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:31.675693    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f93428-2137-428f-ad84-42e9db3e1440-kube-api-access-hscnm" (OuterVolumeSpecName: "kube-api-access-hscnm") pod "e4f93428-2137-428f-ad84-42e9db3e1440" (UID: "e4f93428-2137-428f-ad84-42e9db3e1440"). InnerVolumeSpecName "kube-api-access-hscnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:14:31.675829 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:31.675718    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4f93428-2137-428f-ad84-42e9db3e1440-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e4f93428-2137-428f-ad84-42e9db3e1440" (UID: "e4f93428-2137-428f-ad84-42e9db3e1440"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:14:31.774807 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:31.774682    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hscnm\" (UniqueName: \"kubernetes.io/projected/e4f93428-2137-428f-ad84-42e9db3e1440-kube-api-access-hscnm\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:14:31.774807 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:31.774718    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4f93428-2137-428f-ad84-42e9db3e1440-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:14:31.774807 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:31.774760    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4f93428-2137-428f-ad84-42e9db3e1440-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:14:31.774807 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:31.774773    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4f93428-2137-428f-ad84-42e9db3e1440-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:14:31.774807 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:31.774786    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4f93428-2137-428f-ad84-42e9db3e1440-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:14:32.215804 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:32.215767    2559 generic.go:356] "Generic (PLEG): container finished" podID="e4f93428-2137-428f-ad84-42e9db3e1440" containerID="8aa83e1c49e786eeac6afff962d5d3f3ca49f53e512e3f1b2e7c7316a3af06f8" exitCode=0
Jul 31 15:14:32.216009 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:32.215886    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"
Jul 31 15:14:32.220497 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:32.220467    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8" event={"ID":"e4f93428-2137-428f-ad84-42e9db3e1440","Type":"ContainerDied","Data":"8aa83e1c49e786eeac6afff962d5d3f3ca49f53e512e3f1b2e7c7316a3af06f8"}
Jul 31 15:14:32.220675 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:32.220520    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8" event={"ID":"e4f93428-2137-428f-ad84-42e9db3e1440","Type":"ContainerDied","Data":"9fd17ae8630ac453679283459d1f8e2a52c0571ace9e885fc477760cca54fa82"}
Jul 31 15:14:32.220675 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:32.220547    2559 scope.go:117] "RemoveContainer" containerID="d50ac78236172a780dc8b8e255280c90418c1f39344c2c3f3f03d2a755a0dcd5"
Jul 31 15:14:32.230437 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:32.230416    2559 scope.go:117] "RemoveContainer" containerID="8aa83e1c49e786eeac6afff962d5d3f3ca49f53e512e3f1b2e7c7316a3af06f8"
Jul 31 15:14:32.237912 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:32.237891    2559 scope.go:117] "RemoveContainer" containerID="707b94e084414f5c9d486e1de8dda3dc9d666db542e5010aa14e17cef04da7d0"
Jul 31 15:14:32.241700 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:32.241673    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"]
Jul 31 15:14:32.245835 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:32.245813    2559 scope.go:117] "RemoveContainer" containerID="d50ac78236172a780dc8b8e255280c90418c1f39344c2c3f3f03d2a755a0dcd5"
Jul 31 15:14:32.246101 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:14:32.246082    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d50ac78236172a780dc8b8e255280c90418c1f39344c2c3f3f03d2a755a0dcd5\": container with ID starting with d50ac78236172a780dc8b8e255280c90418c1f39344c2c3f3f03d2a755a0dcd5 not found: ID does not exist" containerID="d50ac78236172a780dc8b8e255280c90418c1f39344c2c3f3f03d2a755a0dcd5"
Jul 31 15:14:32.246149 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:32.246113    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d50ac78236172a780dc8b8e255280c90418c1f39344c2c3f3f03d2a755a0dcd5"} err="failed to get container status \"d50ac78236172a780dc8b8e255280c90418c1f39344c2c3f3f03d2a755a0dcd5\": rpc error: code = NotFound desc = could not find container \"d50ac78236172a780dc8b8e255280c90418c1f39344c2c3f3f03d2a755a0dcd5\": container with ID starting with d50ac78236172a780dc8b8e255280c90418c1f39344c2c3f3f03d2a755a0dcd5 not found: ID does not exist"
Jul 31 15:14:32.246149 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:32.246132    2559 scope.go:117] "RemoveContainer" containerID="8aa83e1c49e786eeac6afff962d5d3f3ca49f53e512e3f1b2e7c7316a3af06f8"
Jul 31 15:14:32.246380 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:14:32.246359    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aa83e1c49e786eeac6afff962d5d3f3ca49f53e512e3f1b2e7c7316a3af06f8\": container with ID starting with 8aa83e1c49e786eeac6afff962d5d3f3ca49f53e512e3f1b2e7c7316a3af06f8 not found: ID does not exist" containerID="8aa83e1c49e786eeac6afff962d5d3f3ca49f53e512e3f1b2e7c7316a3af06f8"
Jul 31 15:14:32.246476 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:32.246384    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aa83e1c49e786eeac6afff962d5d3f3ca49f53e512e3f1b2e7c7316a3af06f8"} err="failed to get container status \"8aa83e1c49e786eeac6afff962d5d3f3ca49f53e512e3f1b2e7c7316a3af06f8\": rpc error: code = NotFound desc = could not find container \"8aa83e1c49e786eeac6afff962d5d3f3ca49f53e512e3f1b2e7c7316a3af06f8\": container with ID starting with 8aa83e1c49e786eeac6afff962d5d3f3ca49f53e512e3f1b2e7c7316a3af06f8 not found: ID does not exist"
Jul 31 15:14:32.246476 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:32.246400    2559 scope.go:117] "RemoveContainer" containerID="707b94e084414f5c9d486e1de8dda3dc9d666db542e5010aa14e17cef04da7d0"
Jul 31 15:14:32.246944 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:14:32.246850    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"707b94e084414f5c9d486e1de8dda3dc9d666db542e5010aa14e17cef04da7d0\": container with ID starting with 707b94e084414f5c9d486e1de8dda3dc9d666db542e5010aa14e17cef04da7d0 not found: ID does not exist" containerID="707b94e084414f5c9d486e1de8dda3dc9d666db542e5010aa14e17cef04da7d0"
Jul 31 15:14:32.246944 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:32.246879    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"707b94e084414f5c9d486e1de8dda3dc9d666db542e5010aa14e17cef04da7d0"} err="failed to get container status \"707b94e084414f5c9d486e1de8dda3dc9d666db542e5010aa14e17cef04da7d0\": rpc error: code = NotFound desc = could not find container \"707b94e084414f5c9d486e1de8dda3dc9d666db542e5010aa14e17cef04da7d0\": container with ID starting with 707b94e084414f5c9d486e1de8dda3dc9d666db542e5010aa14e17cef04da7d0 not found: ID does not exist"
Jul 31 15:14:32.248641 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:32.248622    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-7tfj8"]
Jul 31 15:14:34.221846 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:34.221813    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4f93428-2137-428f-ad84-42e9db3e1440" path="/var/lib/kubelet/pods/e4f93428-2137-428f-ad84-42e9db3e1440/volumes"
Jul 31 15:14:36.975068 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:36.975022    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz" podUID="115552c7-ff25-4bee-9a28-5a0093a60336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 31 15:14:46.976640 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:46.976602    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"
Jul 31 15:14:54.543058 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:54.543027    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"]
Jul 31 15:14:54.543443 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:54.543249    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz" podUID="115552c7-ff25-4bee-9a28-5a0093a60336" containerName="kserve-container" containerID="cri-o://86bfe11760ccdfd20bfd54c35ac4c9e9676a80e10504e791a1d39a03a7a632b9" gracePeriod=30
Jul 31 15:14:54.543443 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:54.543279    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz" podUID="115552c7-ff25-4bee-9a28-5a0093a60336" containerName="kube-rbac-proxy" containerID="cri-o://7896c5e5a5b30bf2593deb1e5d65aec6d3e035e983ac193a54c0404385a44097" gracePeriod=30
Jul 31 15:14:55.298886 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:55.298850    2559 generic.go:356] "Generic (PLEG): container finished" podID="115552c7-ff25-4bee-9a28-5a0093a60336" containerID="7896c5e5a5b30bf2593deb1e5d65aec6d3e035e983ac193a54c0404385a44097" exitCode=2
Jul 31 15:14:55.299144 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:55.298922    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz" event={"ID":"115552c7-ff25-4bee-9a28-5a0093a60336","Type":"ContainerDied","Data":"7896c5e5a5b30bf2593deb1e5d65aec6d3e035e983ac193a54c0404385a44097"}
Jul 31 15:14:56.970190 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:56.970147    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz" podUID="115552c7-ff25-4bee-9a28-5a0093a60336" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 31 15:14:56.975542 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:56.975517    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz" podUID="115552c7-ff25-4bee-9a28-5a0093a60336" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 31 15:14:59.283241 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.283216    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"
Jul 31 15:14:59.315684 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.315651    2559 generic.go:356] "Generic (PLEG): container finished" podID="115552c7-ff25-4bee-9a28-5a0093a60336" containerID="86bfe11760ccdfd20bfd54c35ac4c9e9676a80e10504e791a1d39a03a7a632b9" exitCode=0
Jul 31 15:14:59.315893 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.315689    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz" event={"ID":"115552c7-ff25-4bee-9a28-5a0093a60336","Type":"ContainerDied","Data":"86bfe11760ccdfd20bfd54c35ac4c9e9676a80e10504e791a1d39a03a7a632b9"}
Jul 31 15:14:59.315893 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.315747    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"
Jul 31 15:14:59.315893 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.315779    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz" event={"ID":"115552c7-ff25-4bee-9a28-5a0093a60336","Type":"ContainerDied","Data":"e0bac958b675a9d67841395e8e67a040d6dcf985707f092e8815eed2e51fb8b9"}
Jul 31 15:14:59.315893 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.315817    2559 scope.go:117] "RemoveContainer" containerID="7896c5e5a5b30bf2593deb1e5d65aec6d3e035e983ac193a54c0404385a44097"
Jul 31 15:14:59.324458 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.324439    2559 scope.go:117] "RemoveContainer" containerID="86bfe11760ccdfd20bfd54c35ac4c9e9676a80e10504e791a1d39a03a7a632b9"
Jul 31 15:14:59.331612 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.331595    2559 scope.go:117] "RemoveContainer" containerID="d6c2355a789a65cacbeb671a59246a6d4d5d67b42ecacedcfad4e107822cdd8d"
Jul 31 15:14:59.338547 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.338528    2559 scope.go:117] "RemoveContainer" containerID="7896c5e5a5b30bf2593deb1e5d65aec6d3e035e983ac193a54c0404385a44097"
Jul 31 15:14:59.338843 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:14:59.338825    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7896c5e5a5b30bf2593deb1e5d65aec6d3e035e983ac193a54c0404385a44097\": container with ID starting with 7896c5e5a5b30bf2593deb1e5d65aec6d3e035e983ac193a54c0404385a44097 not found: ID does not exist" containerID="7896c5e5a5b30bf2593deb1e5d65aec6d3e035e983ac193a54c0404385a44097"
Jul 31 15:14:59.338897 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.338855    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7896c5e5a5b30bf2593deb1e5d65aec6d3e035e983ac193a54c0404385a44097"} err="failed to get container status \"7896c5e5a5b30bf2593deb1e5d65aec6d3e035e983ac193a54c0404385a44097\": rpc error: code = NotFound desc = could not find container \"7896c5e5a5b30bf2593deb1e5d65aec6d3e035e983ac193a54c0404385a44097\": container with ID starting with 7896c5e5a5b30bf2593deb1e5d65aec6d3e035e983ac193a54c0404385a44097 not found: ID does not exist"
Jul 31 15:14:59.338897 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.338874    2559 scope.go:117] "RemoveContainer" containerID="86bfe11760ccdfd20bfd54c35ac4c9e9676a80e10504e791a1d39a03a7a632b9"
Jul 31 15:14:59.339113 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:14:59.339097    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86bfe11760ccdfd20bfd54c35ac4c9e9676a80e10504e791a1d39a03a7a632b9\": container with ID starting with 86bfe11760ccdfd20bfd54c35ac4c9e9676a80e10504e791a1d39a03a7a632b9 not found: ID does not exist" containerID="86bfe11760ccdfd20bfd54c35ac4c9e9676a80e10504e791a1d39a03a7a632b9"
Jul 31 15:14:59.339159 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.339118    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86bfe11760ccdfd20bfd54c35ac4c9e9676a80e10504e791a1d39a03a7a632b9"} err="failed to get container status \"86bfe11760ccdfd20bfd54c35ac4c9e9676a80e10504e791a1d39a03a7a632b9\": rpc error: code = NotFound desc = could not find container \"86bfe11760ccdfd20bfd54c35ac4c9e9676a80e10504e791a1d39a03a7a632b9\": container with ID starting with 86bfe11760ccdfd20bfd54c35ac4c9e9676a80e10504e791a1d39a03a7a632b9 not found: ID does not exist"
Jul 31 15:14:59.339159 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.339133    2559 scope.go:117] "RemoveContainer" containerID="d6c2355a789a65cacbeb671a59246a6d4d5d67b42ecacedcfad4e107822cdd8d"
Jul 31 15:14:59.339357 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:14:59.339342    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6c2355a789a65cacbeb671a59246a6d4d5d67b42ecacedcfad4e107822cdd8d\": container with ID starting with d6c2355a789a65cacbeb671a59246a6d4d5d67b42ecacedcfad4e107822cdd8d not found: ID does not exist" containerID="d6c2355a789a65cacbeb671a59246a6d4d5d67b42ecacedcfad4e107822cdd8d"
Jul 31 15:14:59.339400 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.339361    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6c2355a789a65cacbeb671a59246a6d4d5d67b42ecacedcfad4e107822cdd8d"} err="failed to get container status \"d6c2355a789a65cacbeb671a59246a6d4d5d67b42ecacedcfad4e107822cdd8d\": rpc error: code = NotFound desc = could not find container \"d6c2355a789a65cacbeb671a59246a6d4d5d67b42ecacedcfad4e107822cdd8d\": container with ID starting with d6c2355a789a65cacbeb671a59246a6d4d5d67b42ecacedcfad4e107822cdd8d not found: ID does not exist"
Jul 31 15:14:59.391255 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.391226    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r94zs\" (UniqueName: \"kubernetes.io/projected/115552c7-ff25-4bee-9a28-5a0093a60336-kube-api-access-r94zs\") pod \"115552c7-ff25-4bee-9a28-5a0093a60336\" (UID: \"115552c7-ff25-4bee-9a28-5a0093a60336\") "
Jul 31 15:14:59.391368 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.391299    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/115552c7-ff25-4bee-9a28-5a0093a60336-proxy-tls\") pod \"115552c7-ff25-4bee-9a28-5a0093a60336\" (UID: \"115552c7-ff25-4bee-9a28-5a0093a60336\") "
Jul 31 15:14:59.391368 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.391328    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/115552c7-ff25-4bee-9a28-5a0093a60336-kserve-provision-location\") pod \"115552c7-ff25-4bee-9a28-5a0093a60336\" (UID: \"115552c7-ff25-4bee-9a28-5a0093a60336\") "
Jul 31 15:14:59.391368 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.391362    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/115552c7-ff25-4bee-9a28-5a0093a60336-cabundle-cert\") pod \"115552c7-ff25-4bee-9a28-5a0093a60336\" (UID: \"115552c7-ff25-4bee-9a28-5a0093a60336\") "
Jul 31 15:14:59.391495 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.391385    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/115552c7-ff25-4bee-9a28-5a0093a60336-kube-rbac-proxy-sar-config\") pod \"115552c7-ff25-4bee-9a28-5a0093a60336\" (UID: \"115552c7-ff25-4bee-9a28-5a0093a60336\") "
Jul 31 15:14:59.391684 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.391659    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/115552c7-ff25-4bee-9a28-5a0093a60336-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "115552c7-ff25-4bee-9a28-5a0093a60336" (UID: "115552c7-ff25-4bee-9a28-5a0093a60336"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:14:59.391779 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.391680    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/115552c7-ff25-4bee-9a28-5a0093a60336-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "115552c7-ff25-4bee-9a28-5a0093a60336" (UID: "115552c7-ff25-4bee-9a28-5a0093a60336"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:14:59.391779 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.391705    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/115552c7-ff25-4bee-9a28-5a0093a60336-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "115552c7-ff25-4bee-9a28-5a0093a60336" (UID: "115552c7-ff25-4bee-9a28-5a0093a60336"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:14:59.393429 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.393411    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/115552c7-ff25-4bee-9a28-5a0093a60336-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "115552c7-ff25-4bee-9a28-5a0093a60336" (UID: "115552c7-ff25-4bee-9a28-5a0093a60336"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:14:59.393525 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.393509    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115552c7-ff25-4bee-9a28-5a0093a60336-kube-api-access-r94zs" (OuterVolumeSpecName: "kube-api-access-r94zs") pod "115552c7-ff25-4bee-9a28-5a0093a60336" (UID: "115552c7-ff25-4bee-9a28-5a0093a60336"). InnerVolumeSpecName "kube-api-access-r94zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:14:59.492092 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.492056    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/115552c7-ff25-4bee-9a28-5a0093a60336-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:14:59.492092 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.492087    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/115552c7-ff25-4bee-9a28-5a0093a60336-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:14:59.492092 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.492100    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/115552c7-ff25-4bee-9a28-5a0093a60336-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:14:59.492325 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.492112    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/115552c7-ff25-4bee-9a28-5a0093a60336-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:14:59.492325 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.492125    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r94zs\" (UniqueName: \"kubernetes.io/projected/115552c7-ff25-4bee-9a28-5a0093a60336-kube-api-access-r94zs\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:14:59.636826 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.636799    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"]
Jul 31 15:14:59.643911 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.643878    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f8888fcd7-r8qbz"]
Jul 31 15:14:59.729098 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.729061    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"]
Jul 31 15:14:59.729381 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.729367    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="115552c7-ff25-4bee-9a28-5a0093a60336" containerName="kube-rbac-proxy"
Jul 31 15:14:59.729427 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.729384    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="115552c7-ff25-4bee-9a28-5a0093a60336" containerName="kube-rbac-proxy"
Jul 31 15:14:59.729427 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.729400    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="115552c7-ff25-4bee-9a28-5a0093a60336" containerName="storage-initializer"
Jul 31 15:14:59.729427 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.729406    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="115552c7-ff25-4bee-9a28-5a0093a60336" containerName="storage-initializer"
Jul 31 15:14:59.729427 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.729413    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4f93428-2137-428f-ad84-42e9db3e1440" containerName="storage-initializer"
Jul 31 15:14:59.729427 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.729420    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4f93428-2137-428f-ad84-42e9db3e1440" containerName="storage-initializer"
Jul 31 15:14:59.729427 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.729425    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="115552c7-ff25-4bee-9a28-5a0093a60336" containerName="kserve-container"
Jul 31 15:14:59.729427 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.729430    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="115552c7-ff25-4bee-9a28-5a0093a60336" containerName="kserve-container"
Jul 31 15:14:59.729637 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.729436    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4f93428-2137-428f-ad84-42e9db3e1440" containerName="kube-rbac-proxy"
Jul 31 15:14:59.729637 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.729441    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4f93428-2137-428f-ad84-42e9db3e1440" containerName="kube-rbac-proxy"
Jul 31 15:14:59.729637 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.729449    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4f93428-2137-428f-ad84-42e9db3e1440" containerName="kserve-container"
Jul 31 15:14:59.729637 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.729454    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4f93428-2137-428f-ad84-42e9db3e1440" containerName="kserve-container"
Jul 31 15:14:59.729637 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.729499    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="115552c7-ff25-4bee-9a28-5a0093a60336" containerName="kube-rbac-proxy"
Jul 31 15:14:59.729637 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.729506    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4f93428-2137-428f-ad84-42e9db3e1440" containerName="kserve-container"
Jul 31 15:14:59.729637 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.729513    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4f93428-2137-428f-ad84-42e9db3e1440" containerName="kube-rbac-proxy"
Jul 31 15:14:59.729637 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.729520    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="115552c7-ff25-4bee-9a28-5a0093a60336" containerName="kserve-container"
Jul 31 15:14:59.733768 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.733747    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"
Jul 31 15:14:59.736311 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.736279    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 15:14:59.736311 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.736301    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:14:59.736510 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.736317    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Jul 31 15:14:59.736510 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.736352    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 15:14:59.736510 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.736449    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dd67w\""
Jul 31 15:14:59.736510 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.736452    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 31 15:14:59.736644 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.736586    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 15:14:59.743407 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.743388    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"]
Jul 31 15:14:59.794323 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.794289    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5\" (UID: \"d2e6aa46-a3e9-4993-be7d-00fdec10e75a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"
Jul 31 15:14:59.794479 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.794341    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5\" (UID: \"d2e6aa46-a3e9-4993-be7d-00fdec10e75a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"
Jul 31 15:14:59.794479 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.794366    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cznw7\" (UniqueName: \"kubernetes.io/projected/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-kube-api-access-cznw7\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5\" (UID: \"d2e6aa46-a3e9-4993-be7d-00fdec10e75a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"
Jul 31 15:14:59.794479 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.794426    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5\" (UID: \"d2e6aa46-a3e9-4993-be7d-00fdec10e75a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"
Jul 31 15:14:59.794479 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.794474    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5\" (UID: \"d2e6aa46-a3e9-4993-be7d-00fdec10e75a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"
Jul 31 15:14:59.895714 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.895609    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5\" (UID: \"d2e6aa46-a3e9-4993-be7d-00fdec10e75a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"
Jul 31 15:14:59.895714 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.895652    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cznw7\" (UniqueName: \"kubernetes.io/projected/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-kube-api-access-cznw7\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5\" (UID: \"d2e6aa46-a3e9-4993-be7d-00fdec10e75a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"
Jul 31 15:14:59.895714 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.895674    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5\" (UID: \"d2e6aa46-a3e9-4993-be7d-00fdec10e75a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"
Jul 31 15:14:59.895714 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.895690    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5\" (UID: \"d2e6aa46-a3e9-4993-be7d-00fdec10e75a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"
Jul 31 15:14:59.896085 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:14:59.895805    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-serving-cert: secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Jul 31 15:14:59.896085 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.895849    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5\" (UID: \"d2e6aa46-a3e9-4993-be7d-00fdec10e75a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"
Jul 31 15:14:59.896085 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:14:59.895870    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-proxy-tls podName:d2e6aa46-a3e9-4993-be7d-00fdec10e75a nodeName:}" failed. No retries permitted until 2026-07-31 15:15:00.395852975 +0000 UTC m=+1596.707410591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-proxy-tls") pod "sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5" (UID: "d2e6aa46-a3e9-4993-be7d-00fdec10e75a") : secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Jul 31 15:14:59.896245 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.896169    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5\" (UID: \"d2e6aa46-a3e9-4993-be7d-00fdec10e75a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"
Jul 31 15:14:59.896376 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.896356    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5\" (UID: \"d2e6aa46-a3e9-4993-be7d-00fdec10e75a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"
Jul 31 15:14:59.896641 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.896617    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5\" (UID: \"d2e6aa46-a3e9-4993-be7d-00fdec10e75a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"
Jul 31 15:14:59.904105 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:14:59.904084    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cznw7\" (UniqueName: \"kubernetes.io/projected/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-kube-api-access-cznw7\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5\" (UID: \"d2e6aa46-a3e9-4993-be7d-00fdec10e75a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"
Jul 31 15:15:00.220799 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:15:00.220704    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="115552c7-ff25-4bee-9a28-5a0093a60336" path="/var/lib/kubelet/pods/115552c7-ff25-4bee-9a28-5a0093a60336/volumes"
Jul 31 15:15:00.400066 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:15:00.400027    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5\" (UID: \"d2e6aa46-a3e9-4993-be7d-00fdec10e75a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"
Jul 31 15:15:00.402702 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:15:00.402679    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5\" (UID: \"d2e6aa46-a3e9-4993-be7d-00fdec10e75a\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"
Jul 31 15:15:00.644659 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:15:00.644606    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"
Jul 31 15:15:00.774893 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:15:00.774859    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"]
Jul 31 15:15:00.777996 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:15:00.777967    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e6aa46_a3e9_4993_be7d_00fdec10e75a.slice/crio-d5e80dae2a8aa025172168211a74529f407c26ed190c15cbdbdb755907e15b8f WatchSource:0}: Error finding container d5e80dae2a8aa025172168211a74529f407c26ed190c15cbdbdb755907e15b8f: Status 404 returned error can't find the container with id d5e80dae2a8aa025172168211a74529f407c26ed190c15cbdbdb755907e15b8f
Jul 31 15:15:01.326372 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:15:01.326337    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5" event={"ID":"d2e6aa46-a3e9-4993-be7d-00fdec10e75a","Type":"ContainerStarted","Data":"6e8eba79ce0670793168dad6d8f9149d121a0e53610a6f4b0c2ae7558a6e07da"}
Jul 31 15:15:01.326551 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:15:01.326396    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5" event={"ID":"d2e6aa46-a3e9-4993-be7d-00fdec10e75a","Type":"ContainerStarted","Data":"d5e80dae2a8aa025172168211a74529f407c26ed190c15cbdbdb755907e15b8f"}
Jul 31 15:15:05.342250 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:15:05.342203    2559 generic.go:356] "Generic (PLEG): container finished" podID="d2e6aa46-a3e9-4993-be7d-00fdec10e75a" containerID="6e8eba79ce0670793168dad6d8f9149d121a0e53610a6f4b0c2ae7558a6e07da" exitCode=0
Jul 31 15:15:05.342652 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:15:05.342265    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5" event={"ID":"d2e6aa46-a3e9-4993-be7d-00fdec10e75a","Type":"ContainerDied","Data":"6e8eba79ce0670793168dad6d8f9149d121a0e53610a6f4b0c2ae7558a6e07da"}
Jul 31 15:15:05.343342 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:15:05.343326    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 15:17:12.884331 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:12.884292    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5" event={"ID":"d2e6aa46-a3e9-4993-be7d-00fdec10e75a","Type":"ContainerStarted","Data":"413fc5306672f5b01ead31fa999107293a31f227f4470e5a72f84cacda4e288a"}
Jul 31 15:17:12.884765 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:12.884375    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5" event={"ID":"d2e6aa46-a3e9-4993-be7d-00fdec10e75a","Type":"ContainerStarted","Data":"1155d77d38987774c0ebf7257e56c6b33bbe6d76e82560584ff971d9263c8cd6"}
Jul 31 15:17:12.884765 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:12.884590    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"
Jul 31 15:17:12.884765 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:12.884647    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"
Jul 31 15:17:12.905500 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:12.905307    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5" podStartSLOduration=3.30931951 podStartE2EDuration="2m13.905286703s" totalImagesPullingTime="2m6.629929793s" totalInitContainerRuntime="3.9660374s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:17:12.904463234 +0000 UTC m=+1729.216020856" watchObservedRunningTime="2026-07-31 15:17:12.905286703 +0000 UTC m=+1729.216844326"
Jul 31 15:17:18.894004 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:18.893976    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"
Jul 31 15:17:25.979140 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:25.979105    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"]
Jul 31 15:17:25.981748 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:25.981708    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"
Jul 31 15:17:25.982495 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:25.982475    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"]
Jul 31 15:17:25.984340 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:25.984316    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Jul 31 15:17:25.984461 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:25.984347    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Jul 31 15:17:25.984949 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:25.984934    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"
Jul 31 15:17:25.987456 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:25.987436    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Jul 31 15:17:26.000515 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.000488    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"]
Jul 31 15:17:26.003331 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.003302    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"]
Jul 31 15:17:26.023163 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.023126    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-kserve-provision-location\") pod \"isvc-canary-create-predictor-5cd5878b66-ptnsh\" (UID: \"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"
Jul 31 15:17:26.023321 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.023262    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-cabundle-cert\") pod \"isvc-canary-create-predictor-5cd5878b66-ptnsh\" (UID: \"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"
Jul 31 15:17:26.023321 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.023298    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vp5c\" (UniqueName: \"kubernetes.io/projected/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-kube-api-access-9vp5c\") pod \"isvc-canary-create-predictor-5cd5878b66-ptnsh\" (UID: \"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"
Jul 31 15:17:26.023435 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.023375    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-proxy-tls\") pod \"isvc-canary-create-predictor-5cd5878b66-ptnsh\" (UID: \"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"
Jul 31 15:17:26.023435 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.023403    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5cd5878b66-ptnsh\" (UID: \"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"
Jul 31 15:17:26.124142 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.124101    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37e06607-0588-4830-9530-612d93b90817-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5f99649479-bnz2w\" (UID: \"37e06607-0588-4830-9530-612d93b90817\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"
Jul 31 15:17:26.124142 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.124140    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crxzz\" (UniqueName: \"kubernetes.io/projected/37e06607-0588-4830-9530-612d93b90817-kube-api-access-crxzz\") pod \"isvc-canary-create-v2-predictor-5f99649479-bnz2w\" (UID: \"37e06607-0588-4830-9530-612d93b90817\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"
Jul 31 15:17:26.124383 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.124162    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-kserve-provision-location\") pod \"isvc-canary-create-predictor-5cd5878b66-ptnsh\" (UID: \"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"
Jul 31 15:17:26.124383 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.124190    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-cabundle-cert\") pod \"isvc-canary-create-predictor-5cd5878b66-ptnsh\" (UID: \"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"
Jul 31 15:17:26.124383 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.124220    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vp5c\" (UniqueName: \"kubernetes.io/projected/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-kube-api-access-9vp5c\") pod \"isvc-canary-create-predictor-5cd5878b66-ptnsh\" (UID: \"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"
Jul 31 15:17:26.124383 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.124341    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37e06607-0588-4830-9530-612d93b90817-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5f99649479-bnz2w\" (UID: \"37e06607-0588-4830-9530-612d93b90817\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"
Jul 31 15:17:26.124589 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.124385    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37e06607-0588-4830-9530-612d93b90817-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5f99649479-bnz2w\" (UID: \"37e06607-0588-4830-9530-612d93b90817\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"
Jul 31 15:17:26.124589 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.124426    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-proxy-tls\") pod \"isvc-canary-create-predictor-5cd5878b66-ptnsh\" (UID: \"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"
Jul 31 15:17:26.124589 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.124453    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5cd5878b66-ptnsh\" (UID: \"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"
Jul 31 15:17:26.124589 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:17:26.124535    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Jul 31 15:17:26.124589 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.124573    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37e06607-0588-4830-9530-612d93b90817-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5f99649479-bnz2w\" (UID: \"37e06607-0588-4830-9530-612d93b90817\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"
Jul 31 15:17:26.124806 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.124574    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-kserve-provision-location\") pod \"isvc-canary-create-predictor-5cd5878b66-ptnsh\" (UID: \"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"
Jul 31 15:17:26.124806 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:17:26.124612    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-proxy-tls podName:c8930d19-1ea2-4fd8-88b0-77edd73d4d6c nodeName:}" failed. No retries permitted until 2026-07-31 15:17:26.624589908 +0000 UTC m=+1742.936147511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-proxy-tls") pod "isvc-canary-create-predictor-5cd5878b66-ptnsh" (UID: "c8930d19-1ea2-4fd8-88b0-77edd73d4d6c") : secret "isvc-canary-create-predictor-serving-cert" not found
Jul 31 15:17:26.124897 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.124879    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-cabundle-cert\") pod \"isvc-canary-create-predictor-5cd5878b66-ptnsh\" (UID: \"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"
Jul 31 15:17:26.125113 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.125096    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5cd5878b66-ptnsh\" (UID: \"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"
Jul 31 15:17:26.143620 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.143585    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vp5c\" (UniqueName: \"kubernetes.io/projected/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-kube-api-access-9vp5c\") pod \"isvc-canary-create-predictor-5cd5878b66-ptnsh\" (UID: \"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"
Jul 31 15:17:26.225720 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.225679    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37e06607-0588-4830-9530-612d93b90817-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5f99649479-bnz2w\" (UID: \"37e06607-0588-4830-9530-612d93b90817\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"
Jul 31 15:17:26.225923 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.225722    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37e06607-0588-4830-9530-612d93b90817-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5f99649479-bnz2w\" (UID: \"37e06607-0588-4830-9530-612d93b90817\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"
Jul 31 15:17:26.225923 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.225852    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crxzz\" (UniqueName: \"kubernetes.io/projected/37e06607-0588-4830-9530-612d93b90817-kube-api-access-crxzz\") pod \"isvc-canary-create-v2-predictor-5f99649479-bnz2w\" (UID: \"37e06607-0588-4830-9530-612d93b90817\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"
Jul 31 15:17:26.226046 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.225925    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37e06607-0588-4830-9530-612d93b90817-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5f99649479-bnz2w\" (UID: \"37e06607-0588-4830-9530-612d93b90817\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"
Jul 31 15:17:26.226046 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.225947    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37e06607-0588-4830-9530-612d93b90817-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5f99649479-bnz2w\" (UID: \"37e06607-0588-4830-9530-612d93b90817\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"
Jul 31 15:17:26.226391 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.226367    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37e06607-0588-4830-9530-612d93b90817-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5f99649479-bnz2w\" (UID: \"37e06607-0588-4830-9530-612d93b90817\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"
Jul 31 15:17:26.226479 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.226461    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37e06607-0588-4830-9530-612d93b90817-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5f99649479-bnz2w\" (UID: \"37e06607-0588-4830-9530-612d93b90817\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"
Jul 31 15:17:26.226535 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.226515    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37e06607-0588-4830-9530-612d93b90817-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5f99649479-bnz2w\" (UID: \"37e06607-0588-4830-9530-612d93b90817\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"
Jul 31 15:17:26.228472 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.228452    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37e06607-0588-4830-9530-612d93b90817-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5f99649479-bnz2w\" (UID: \"37e06607-0588-4830-9530-612d93b90817\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"
Jul 31 15:17:26.235349 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.235293    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crxzz\" (UniqueName: \"kubernetes.io/projected/37e06607-0588-4830-9530-612d93b90817-kube-api-access-crxzz\") pod \"isvc-canary-create-v2-predictor-5f99649479-bnz2w\" (UID: \"37e06607-0588-4830-9530-612d93b90817\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"
Jul 31 15:17:26.301338 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.301296    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"
Jul 31 15:17:26.470619 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.470584    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"]
Jul 31 15:17:26.473761 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:17:26.473715    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37e06607_0588_4830_9530_612d93b90817.slice/crio-ed43e237e5b1547cf1bb8ee08391b4c8a9d72d48ca2b0b96b4e5da5da8721e70 WatchSource:0}: Error finding container ed43e237e5b1547cf1bb8ee08391b4c8a9d72d48ca2b0b96b4e5da5da8721e70: Status 404 returned error can't find the container with id ed43e237e5b1547cf1bb8ee08391b4c8a9d72d48ca2b0b96b4e5da5da8721e70
Jul 31 15:17:26.628981 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.628928    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-proxy-tls\") pod \"isvc-canary-create-predictor-5cd5878b66-ptnsh\" (UID: \"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"
Jul 31 15:17:26.631458 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.631439    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-proxy-tls\") pod \"isvc-canary-create-predictor-5cd5878b66-ptnsh\" (UID: \"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"
Jul 31 15:17:26.893512 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.893468    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"
Jul 31 15:17:26.934889 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.934850    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w" event={"ID":"37e06607-0588-4830-9530-612d93b90817","Type":"ContainerStarted","Data":"71c9a11bffea99137f25cef8b9ecf816bb34955390b57a24932cc57ad2da355b"}
Jul 31 15:17:26.935084 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:26.934908    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w" event={"ID":"37e06607-0588-4830-9530-612d93b90817","Type":"ContainerStarted","Data":"ed43e237e5b1547cf1bb8ee08391b4c8a9d72d48ca2b0b96b4e5da5da8721e70"}
Jul 31 15:17:27.206920 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:27.206833    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"]
Jul 31 15:17:27.209152 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:17:27.209127    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8930d19_1ea2_4fd8_88b0_77edd73d4d6c.slice/crio-00b28e2994610cd77f8c7b00f97e2ec2bf5d8512edba7cdc06a1a2e8e5ffe0e4 WatchSource:0}: Error finding container 00b28e2994610cd77f8c7b00f97e2ec2bf5d8512edba7cdc06a1a2e8e5ffe0e4: Status 404 returned error can't find the container with id 00b28e2994610cd77f8c7b00f97e2ec2bf5d8512edba7cdc06a1a2e8e5ffe0e4
Jul 31 15:17:27.940628 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:27.940592    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh" event={"ID":"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c","Type":"ContainerStarted","Data":"037bb01f5cc01be30a363f322ceda0aadcea5fb267790d42b583a04fa7ec1edf"}
Jul 31 15:17:27.940831 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:27.940656    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh" event={"ID":"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c","Type":"ContainerStarted","Data":"00b28e2994610cd77f8c7b00f97e2ec2bf5d8512edba7cdc06a1a2e8e5ffe0e4"}
Jul 31 15:17:31.957285 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:31.957249    2559 generic.go:356] "Generic (PLEG): container finished" podID="37e06607-0588-4830-9530-612d93b90817" containerID="71c9a11bffea99137f25cef8b9ecf816bb34955390b57a24932cc57ad2da355b" exitCode=0
Jul 31 15:17:31.957914 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:31.957334    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w" event={"ID":"37e06607-0588-4830-9530-612d93b90817","Type":"ContainerDied","Data":"71c9a11bffea99137f25cef8b9ecf816bb34955390b57a24932cc57ad2da355b"}
Jul 31 15:17:31.958896 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:31.958870    2559 generic.go:356] "Generic (PLEG): container finished" podID="c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" containerID="037bb01f5cc01be30a363f322ceda0aadcea5fb267790d42b583a04fa7ec1edf" exitCode=0
Jul 31 15:17:31.958995 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:31.958927    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh" event={"ID":"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c","Type":"ContainerDied","Data":"037bb01f5cc01be30a363f322ceda0aadcea5fb267790d42b583a04fa7ec1edf"}
Jul 31 15:17:32.965936 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:32.965894    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh" event={"ID":"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c","Type":"ContainerStarted","Data":"09b46f76e99a6ee0cdff94f90b7ed0f29e8e2a2b77f0d9571a105a50604be37f"}
Jul 31 15:17:32.966437 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:32.965979    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh" event={"ID":"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c","Type":"ContainerStarted","Data":"e5e374d731a423f87cd2f4ede655a22a9675b6ece9e2c007ce5294906a625150"}
Jul 31 15:17:32.967320 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:32.967285    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"
Jul 31 15:17:32.967472 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:32.967381    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"
Jul 31 15:17:32.974520 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:32.973845    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh" podUID="c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 31 15:17:32.977031 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:32.977003    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w" event={"ID":"37e06607-0588-4830-9530-612d93b90817","Type":"ContainerStarted","Data":"7213ee272490ab50c492d22bd076b4a1c2cfbdefe15c32e840abd5e493d07fe5"}
Jul 31 15:17:32.977200 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:32.977094    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w" event={"ID":"37e06607-0588-4830-9530-612d93b90817","Type":"ContainerStarted","Data":"0e7ed24e138cceb6da591fda60cda751824698e2d64bdf5187556a39a632ef18"}
Jul 31 15:17:32.977398 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:32.977372    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"
Jul 31 15:17:32.977529 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:32.977430    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"
Jul 31 15:17:32.978618 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:32.978596    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w" podUID="37e06607-0588-4830-9530-612d93b90817" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 31 15:17:32.992146 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:32.992082    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh" podStartSLOduration=4.005814081 podStartE2EDuration="7.992068038s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.986253957s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:17:32.991024382 +0000 UTC m=+1749.302582004" watchObservedRunningTime="2026-07-31 15:17:32.992068038 +0000 UTC m=+1749.303625660"
Jul 31 15:17:33.013391 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:33.013338    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w" podStartSLOduration=3.447858645 podStartE2EDuration="8.013323369s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.565464724s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:17:33.012535491 +0000 UTC m=+1749.324093123" watchObservedRunningTime="2026-07-31 15:17:33.013323369 +0000 UTC m=+1749.324880991"
Jul 31 15:17:33.981086 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:33.981048    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh" podUID="c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 31 15:17:33.981461 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:33.981056    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w" podUID="37e06607-0588-4830-9530-612d93b90817" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 31 15:17:38.985992 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:38.985954    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"
Jul 31 15:17:38.986403 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:38.986349    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"
Jul 31 15:17:38.986463 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:38.986435    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh" podUID="c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 31 15:17:38.986944 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:38.986914    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w" podUID="37e06607-0588-4830-9530-612d93b90817" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 31 15:17:48.967555 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:48.967500    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5" podUID="d2e6aa46-a3e9-4993-be7d-00fdec10e75a" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 31 15:17:48.987065 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:48.987029    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh" podUID="c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 31 15:17:48.987246 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:48.987029    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w" podUID="37e06607-0588-4830-9530-612d93b90817" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 31 15:17:58.897026 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:58.896991    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"
Jul 31 15:17:58.986768 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:58.986710    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh" podUID="c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 31 15:17:58.986941 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:58.986868    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w" podUID="37e06607-0588-4830-9530-612d93b90817" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 31 15:17:59.901851 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:59.901817    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"]
Jul 31 15:17:59.902307 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:59.902037    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5" podUID="d2e6aa46-a3e9-4993-be7d-00fdec10e75a" containerName="kserve-container" containerID="cri-o://1155d77d38987774c0ebf7257e56c6b33bbe6d76e82560584ff971d9263c8cd6" gracePeriod=30
Jul 31 15:17:59.902307 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:17:59.902093    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5" podUID="d2e6aa46-a3e9-4993-be7d-00fdec10e75a" containerName="kube-rbac-proxy" containerID="cri-o://413fc5306672f5b01ead31fa999107293a31f227f4470e5a72f84cacda4e288a" gracePeriod=30
Jul 31 15:18:00.087961 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:00.087928    2559 generic.go:356] "Generic (PLEG): container finished" podID="d2e6aa46-a3e9-4993-be7d-00fdec10e75a" containerID="413fc5306672f5b01ead31fa999107293a31f227f4470e5a72f84cacda4e288a" exitCode=2
Jul 31 15:18:00.088147 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:00.088009    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5" event={"ID":"d2e6aa46-a3e9-4993-be7d-00fdec10e75a","Type":"ContainerDied","Data":"413fc5306672f5b01ead31fa999107293a31f227f4470e5a72f84cacda4e288a"}
Jul 31 15:18:03.888489 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:03.888441    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5" podUID="d2e6aa46-a3e9-4993-be7d-00fdec10e75a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 31 15:18:05.050504 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:05.050469    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"]
Jul 31 15:18:05.053068 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:05.053047    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"
Jul 31 15:18:05.057624 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:05.057600    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Jul 31 15:18:05.058144 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:05.058123    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Jul 31 15:18:05.070094 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:05.070065    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"]
Jul 31 15:18:05.144714 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:05.144674    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l44t\" (UniqueName: \"kubernetes.io/projected/01b61a33-d6ea-459e-af61-acb0728cc881-kube-api-access-2l44t\") pod \"isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9\" (UID: \"01b61a33-d6ea-459e-af61-acb0728cc881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"
Jul 31 15:18:05.144944 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:05.144752    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01b61a33-d6ea-459e-af61-acb0728cc881-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9\" (UID: \"01b61a33-d6ea-459e-af61-acb0728cc881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"
Jul 31 15:18:05.144944 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:05.144787    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01b61a33-d6ea-459e-af61-acb0728cc881-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9\" (UID: \"01b61a33-d6ea-459e-af61-acb0728cc881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"
Jul 31 15:18:05.144944 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:05.144821    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01b61a33-d6ea-459e-af61-acb0728cc881-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9\" (UID: \"01b61a33-d6ea-459e-af61-acb0728cc881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"
Jul 31 15:18:05.144944 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:05.144848    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01b61a33-d6ea-459e-af61-acb0728cc881-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9\" (UID: \"01b61a33-d6ea-459e-af61-acb0728cc881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"
Jul 31 15:18:05.245974 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:05.245938    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l44t\" (UniqueName: \"kubernetes.io/projected/01b61a33-d6ea-459e-af61-acb0728cc881-kube-api-access-2l44t\") pod \"isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9\" (UID: \"01b61a33-d6ea-459e-af61-acb0728cc881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"
Jul 31 15:18:05.245974 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:05.245979    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01b61a33-d6ea-459e-af61-acb0728cc881-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9\" (UID: \"01b61a33-d6ea-459e-af61-acb0728cc881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"
Jul 31 15:18:05.246242 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:05.246011    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01b61a33-d6ea-459e-af61-acb0728cc881-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9\" (UID: \"01b61a33-d6ea-459e-af61-acb0728cc881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"
Jul 31 15:18:05.246242 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:05.246050    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01b61a33-d6ea-459e-af61-acb0728cc881-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9\" (UID: \"01b61a33-d6ea-459e-af61-acb0728cc881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"
Jul 31 15:18:05.246242 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:05.246074    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01b61a33-d6ea-459e-af61-acb0728cc881-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9\" (UID: \"01b61a33-d6ea-459e-af61-acb0728cc881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"
Jul 31 15:18:05.246242 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:18:05.246205    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-serving-cert: secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Jul 31 15:18:05.246412 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:18:05.246270    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01b61a33-d6ea-459e-af61-acb0728cc881-proxy-tls podName:01b61a33-d6ea-459e-af61-acb0728cc881 nodeName:}" failed. No retries permitted until 2026-07-31 15:18:05.74624793 +0000 UTC m=+1782.057805533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/01b61a33-d6ea-459e-af61-acb0728cc881-proxy-tls") pod "isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9" (UID: "01b61a33-d6ea-459e-af61-acb0728cc881") : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Jul 31 15:18:05.246565 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:05.246538    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01b61a33-d6ea-459e-af61-acb0728cc881-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9\" (UID: \"01b61a33-d6ea-459e-af61-acb0728cc881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"
Jul 31 15:18:05.246677 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:05.246660    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01b61a33-d6ea-459e-af61-acb0728cc881-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9\" (UID: \"01b61a33-d6ea-459e-af61-acb0728cc881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"
Jul 31 15:18:05.246779 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:05.246763    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01b61a33-d6ea-459e-af61-acb0728cc881-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9\" (UID: \"01b61a33-d6ea-459e-af61-acb0728cc881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"
Jul 31 15:18:05.255117 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:05.255090    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l44t\" (UniqueName: \"kubernetes.io/projected/01b61a33-d6ea-459e-af61-acb0728cc881-kube-api-access-2l44t\") pod \"isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9\" (UID: \"01b61a33-d6ea-459e-af61-acb0728cc881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"
Jul 31 15:18:05.750943 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:05.750904    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01b61a33-d6ea-459e-af61-acb0728cc881-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9\" (UID: \"01b61a33-d6ea-459e-af61-acb0728cc881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"
Jul 31 15:18:05.753417 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:05.753397    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01b61a33-d6ea-459e-af61-acb0728cc881-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9\" (UID: \"01b61a33-d6ea-459e-af61-acb0728cc881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"
Jul 31 15:18:05.963402 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:05.963350    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"
Jul 31 15:18:06.102221 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:06.102199    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"]
Jul 31 15:18:06.104858 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:18:06.104834    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b61a33_d6ea_459e_af61_acb0728cc881.slice/crio-d93589b1a4786d2360115432d3f8c57c615dc568f7e032b821f0b974d261042d WatchSource:0}: Error finding container d93589b1a4786d2360115432d3f8c57c615dc568f7e032b821f0b974d261042d: Status 404 returned error can't find the container with id d93589b1a4786d2360115432d3f8c57c615dc568f7e032b821f0b974d261042d
Jul 31 15:18:06.113664 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:06.113633    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9" event={"ID":"01b61a33-d6ea-459e-af61-acb0728cc881","Type":"ContainerStarted","Data":"d93589b1a4786d2360115432d3f8c57c615dc568f7e032b821f0b974d261042d"}
Jul 31 15:18:07.119230 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:07.119193    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9" event={"ID":"01b61a33-d6ea-459e-af61-acb0728cc881","Type":"ContainerStarted","Data":"9cc896c27b0d3fdb8c5b5405f5aa37ff5bcdd31e837db23b32e6868f7e12be62"}
Jul 31 15:18:08.048705 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.048682    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"
Jul 31 15:18:08.125002 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.124917    2559 generic.go:356] "Generic (PLEG): container finished" podID="d2e6aa46-a3e9-4993-be7d-00fdec10e75a" containerID="1155d77d38987774c0ebf7257e56c6b33bbe6d76e82560584ff971d9263c8cd6" exitCode=0
Jul 31 15:18:08.125002 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.124974    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"
Jul 31 15:18:08.125469 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.125006    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5" event={"ID":"d2e6aa46-a3e9-4993-be7d-00fdec10e75a","Type":"ContainerDied","Data":"1155d77d38987774c0ebf7257e56c6b33bbe6d76e82560584ff971d9263c8cd6"}
Jul 31 15:18:08.125469 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.125072    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5" event={"ID":"d2e6aa46-a3e9-4993-be7d-00fdec10e75a","Type":"ContainerDied","Data":"d5e80dae2a8aa025172168211a74529f407c26ed190c15cbdbdb755907e15b8f"}
Jul 31 15:18:08.125469 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.125103    2559 scope.go:117] "RemoveContainer" containerID="413fc5306672f5b01ead31fa999107293a31f227f4470e5a72f84cacda4e288a"
Jul 31 15:18:08.133240 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.133220    2559 scope.go:117] "RemoveContainer" containerID="1155d77d38987774c0ebf7257e56c6b33bbe6d76e82560584ff971d9263c8cd6"
Jul 31 15:18:08.140901 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.140882    2559 scope.go:117] "RemoveContainer" containerID="6e8eba79ce0670793168dad6d8f9149d121a0e53610a6f4b0c2ae7558a6e07da"
Jul 31 15:18:08.148617 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.148599    2559 scope.go:117] "RemoveContainer" containerID="413fc5306672f5b01ead31fa999107293a31f227f4470e5a72f84cacda4e288a"
Jul 31 15:18:08.148922 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:18:08.148902    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"413fc5306672f5b01ead31fa999107293a31f227f4470e5a72f84cacda4e288a\": container with ID starting with 413fc5306672f5b01ead31fa999107293a31f227f4470e5a72f84cacda4e288a not found: ID does not exist" containerID="413fc5306672f5b01ead31fa999107293a31f227f4470e5a72f84cacda4e288a"
Jul 31 15:18:08.148988 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.148931    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"413fc5306672f5b01ead31fa999107293a31f227f4470e5a72f84cacda4e288a"} err="failed to get container status \"413fc5306672f5b01ead31fa999107293a31f227f4470e5a72f84cacda4e288a\": rpc error: code = NotFound desc = could not find container \"413fc5306672f5b01ead31fa999107293a31f227f4470e5a72f84cacda4e288a\": container with ID starting with 413fc5306672f5b01ead31fa999107293a31f227f4470e5a72f84cacda4e288a not found: ID does not exist"
Jul 31 15:18:08.148988 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.148955    2559 scope.go:117] "RemoveContainer" containerID="1155d77d38987774c0ebf7257e56c6b33bbe6d76e82560584ff971d9263c8cd6"
Jul 31 15:18:08.149159 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:18:08.149140    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1155d77d38987774c0ebf7257e56c6b33bbe6d76e82560584ff971d9263c8cd6\": container with ID starting with 1155d77d38987774c0ebf7257e56c6b33bbe6d76e82560584ff971d9263c8cd6 not found: ID does not exist" containerID="1155d77d38987774c0ebf7257e56c6b33bbe6d76e82560584ff971d9263c8cd6"
Jul 31 15:18:08.149209 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.149170    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1155d77d38987774c0ebf7257e56c6b33bbe6d76e82560584ff971d9263c8cd6"} err="failed to get container status \"1155d77d38987774c0ebf7257e56c6b33bbe6d76e82560584ff971d9263c8cd6\": rpc error: code = NotFound desc = could not find container \"1155d77d38987774c0ebf7257e56c6b33bbe6d76e82560584ff971d9263c8cd6\": container with ID starting with 1155d77d38987774c0ebf7257e56c6b33bbe6d76e82560584ff971d9263c8cd6 not found: ID does not exist"
Jul 31 15:18:08.149209 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.149188    2559 scope.go:117] "RemoveContainer" containerID="6e8eba79ce0670793168dad6d8f9149d121a0e53610a6f4b0c2ae7558a6e07da"
Jul 31 15:18:08.149442 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:18:08.149425    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e8eba79ce0670793168dad6d8f9149d121a0e53610a6f4b0c2ae7558a6e07da\": container with ID starting with 6e8eba79ce0670793168dad6d8f9149d121a0e53610a6f4b0c2ae7558a6e07da not found: ID does not exist" containerID="6e8eba79ce0670793168dad6d8f9149d121a0e53610a6f4b0c2ae7558a6e07da"
Jul 31 15:18:08.149502 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.149445    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e8eba79ce0670793168dad6d8f9149d121a0e53610a6f4b0c2ae7558a6e07da"} err="failed to get container status \"6e8eba79ce0670793168dad6d8f9149d121a0e53610a6f4b0c2ae7558a6e07da\": rpc error: code = NotFound desc = could not find container \"6e8eba79ce0670793168dad6d8f9149d121a0e53610a6f4b0c2ae7558a6e07da\": container with ID starting with 6e8eba79ce0670793168dad6d8f9149d121a0e53610a6f4b0c2ae7558a6e07da not found: ID does not exist"
Jul 31 15:18:08.173812 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.173783    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cznw7\" (UniqueName: \"kubernetes.io/projected/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-kube-api-access-cznw7\") pod \"d2e6aa46-a3e9-4993-be7d-00fdec10e75a\" (UID: \"d2e6aa46-a3e9-4993-be7d-00fdec10e75a\") "
Jul 31 15:18:08.173970 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.173846    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-kserve-provision-location\") pod \"d2e6aa46-a3e9-4993-be7d-00fdec10e75a\" (UID: \"d2e6aa46-a3e9-4993-be7d-00fdec10e75a\") "
Jul 31 15:18:08.173970 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.173890    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-cabundle-cert\") pod \"d2e6aa46-a3e9-4993-be7d-00fdec10e75a\" (UID: \"d2e6aa46-a3e9-4993-be7d-00fdec10e75a\") "
Jul 31 15:18:08.173970 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.173910    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-proxy-tls\") pod \"d2e6aa46-a3e9-4993-be7d-00fdec10e75a\" (UID: \"d2e6aa46-a3e9-4993-be7d-00fdec10e75a\") "
Jul 31 15:18:08.173970 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.173938    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-kube-rbac-proxy-sar-config\") pod \"d2e6aa46-a3e9-4993-be7d-00fdec10e75a\" (UID: \"d2e6aa46-a3e9-4993-be7d-00fdec10e75a\") "
Jul 31 15:18:08.174310 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.174280    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d2e6aa46-a3e9-4993-be7d-00fdec10e75a" (UID: "d2e6aa46-a3e9-4993-be7d-00fdec10e75a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:18:08.174409 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.174310    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d2e6aa46-a3e9-4993-be7d-00fdec10e75a" (UID: "d2e6aa46-a3e9-4993-be7d-00fdec10e75a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:18:08.174409 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.174351    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d2e6aa46-a3e9-4993-be7d-00fdec10e75a" (UID: "d2e6aa46-a3e9-4993-be7d-00fdec10e75a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:18:08.176716 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.176692    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-kube-api-access-cznw7" (OuterVolumeSpecName: "kube-api-access-cznw7") pod "d2e6aa46-a3e9-4993-be7d-00fdec10e75a" (UID: "d2e6aa46-a3e9-4993-be7d-00fdec10e75a"). InnerVolumeSpecName "kube-api-access-cznw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:18:08.176816 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.176692    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d2e6aa46-a3e9-4993-be7d-00fdec10e75a" (UID: "d2e6aa46-a3e9-4993-be7d-00fdec10e75a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:18:08.274754 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.274707    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:18:08.274754 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.274754    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:18:08.275005 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.274768    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:18:08.275005 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.274778    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cznw7\" (UniqueName: \"kubernetes.io/projected/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-kube-api-access-cznw7\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:18:08.275005 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.274788    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2e6aa46-a3e9-4993-be7d-00fdec10e75a-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:18:08.442746 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.442647    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"]
Jul 31 15:18:08.446703 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.446675    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wdnj5"]
Jul 31 15:18:08.986802 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.986762    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w" podUID="37e06607-0588-4830-9530-612d93b90817" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 31 15:18:08.986996 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:08.986768    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh" podUID="c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 31 15:18:10.221976 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:10.221929    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2e6aa46-a3e9-4993-be7d-00fdec10e75a" path="/var/lib/kubelet/pods/d2e6aa46-a3e9-4993-be7d-00fdec10e75a/volumes"
Jul 31 15:18:12.142645 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:12.142611    2559 generic.go:356] "Generic (PLEG): container finished" podID="01b61a33-d6ea-459e-af61-acb0728cc881" containerID="9cc896c27b0d3fdb8c5b5405f5aa37ff5bcdd31e837db23b32e6868f7e12be62" exitCode=0
Jul 31 15:18:12.143081 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:12.142670    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9" event={"ID":"01b61a33-d6ea-459e-af61-acb0728cc881","Type":"ContainerDied","Data":"9cc896c27b0d3fdb8c5b5405f5aa37ff5bcdd31e837db23b32e6868f7e12be62"}
Jul 31 15:18:13.149325 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:13.149288    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9" event={"ID":"01b61a33-d6ea-459e-af61-acb0728cc881","Type":"ContainerStarted","Data":"11d8dbf80774b3433301ef52f046cc817cf1f014341ebf5542f8cdf1ef06f054"}
Jul 31 15:18:13.149777 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:13.149355    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9" event={"ID":"01b61a33-d6ea-459e-af61-acb0728cc881","Type":"ContainerStarted","Data":"7367eb8ccf11fa790d5fc14e4abd964917df169b1cc86a814a5e06e667a53c7b"}
Jul 31 15:18:13.149777 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:13.149571    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"
Jul 31 15:18:13.173241 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:13.173172    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9" podStartSLOduration=2.314122226 podStartE2EDuration="8.173151928s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.859029702s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:18:13.171474199 +0000 UTC m=+1789.483031821" watchObservedRunningTime="2026-07-31 15:18:13.173151928 +0000 UTC m=+1789.484709551"
Jul 31 15:18:14.156706 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:14.156663    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"
Jul 31 15:18:14.159004 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:14.158972    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9" podUID="01b61a33-d6ea-459e-af61-acb0728cc881" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 31 15:18:15.159288 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:15.159200    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9" podUID="01b61a33-d6ea-459e-af61-acb0728cc881" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 31 15:18:18.987170 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:18.987136    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w" podUID="37e06607-0588-4830-9530-612d93b90817" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 31 15:18:18.987557 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:18.987181    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh" podUID="c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 31 15:18:20.164027 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:20.163995    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"
Jul 31 15:18:20.164782 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:20.164754    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9" podUID="01b61a33-d6ea-459e-af61-acb0728cc881" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 31 15:18:28.986657 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:28.986619    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh" podUID="c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 31 15:18:28.987102 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:28.986794    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w" podUID="37e06607-0588-4830-9530-612d93b90817" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 31 15:18:30.165853 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:30.165825    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"
Jul 31 15:18:36.768605 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:36.768574    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9_01b61a33-d6ea-459e-af61-acb0728cc881/kserve-container/0.log"
Jul 31 15:18:36.900760 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:36.900649    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"]
Jul 31 15:18:36.901037 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:36.901009    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9" podUID="01b61a33-d6ea-459e-af61-acb0728cc881" containerName="kserve-container" containerID="cri-o://7367eb8ccf11fa790d5fc14e4abd964917df169b1cc86a814a5e06e667a53c7b" gracePeriod=30
Jul 31 15:18:36.901169 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:36.901020    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9" podUID="01b61a33-d6ea-459e-af61-acb0728cc881" containerName="kube-rbac-proxy" containerID="cri-o://11d8dbf80774b3433301ef52f046cc817cf1f014341ebf5542f8cdf1ef06f054" gracePeriod=30
Jul 31 15:18:37.253133 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:37.253100    2559 generic.go:356] "Generic (PLEG): container finished" podID="01b61a33-d6ea-459e-af61-acb0728cc881" containerID="11d8dbf80774b3433301ef52f046cc817cf1f014341ebf5542f8cdf1ef06f054" exitCode=2
Jul 31 15:18:37.253300 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:37.253143    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9" event={"ID":"01b61a33-d6ea-459e-af61-acb0728cc881","Type":"ContainerDied","Data":"11d8dbf80774b3433301ef52f046cc817cf1f014341ebf5542f8cdf1ef06f054"}
Jul 31 15:18:38.040067 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.040038    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"
Jul 31 15:18:38.121230 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.121197    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01b61a33-d6ea-459e-af61-acb0728cc881-kserve-provision-location\") pod \"01b61a33-d6ea-459e-af61-acb0728cc881\" (UID: \"01b61a33-d6ea-459e-af61-acb0728cc881\") "
Jul 31 15:18:38.121230 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.121246    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01b61a33-d6ea-459e-af61-acb0728cc881-kube-rbac-proxy-sar-config\") pod \"01b61a33-d6ea-459e-af61-acb0728cc881\" (UID: \"01b61a33-d6ea-459e-af61-acb0728cc881\") "
Jul 31 15:18:38.121498 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.121277    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01b61a33-d6ea-459e-af61-acb0728cc881-cabundle-cert\") pod \"01b61a33-d6ea-459e-af61-acb0728cc881\" (UID: \"01b61a33-d6ea-459e-af61-acb0728cc881\") "
Jul 31 15:18:38.121498 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.121310    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2l44t\" (UniqueName: \"kubernetes.io/projected/01b61a33-d6ea-459e-af61-acb0728cc881-kube-api-access-2l44t\") pod \"01b61a33-d6ea-459e-af61-acb0728cc881\" (UID: \"01b61a33-d6ea-459e-af61-acb0728cc881\") "
Jul 31 15:18:38.121498 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.121344    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01b61a33-d6ea-459e-af61-acb0728cc881-proxy-tls\") pod \"01b61a33-d6ea-459e-af61-acb0728cc881\" (UID: \"01b61a33-d6ea-459e-af61-acb0728cc881\") "
Jul 31 15:18:38.121661 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.121580    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b61a33-d6ea-459e-af61-acb0728cc881-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "01b61a33-d6ea-459e-af61-acb0728cc881" (UID: "01b61a33-d6ea-459e-af61-acb0728cc881"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:18:38.121661 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.121634    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b61a33-d6ea-459e-af61-acb0728cc881-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "01b61a33-d6ea-459e-af61-acb0728cc881" (UID: "01b61a33-d6ea-459e-af61-acb0728cc881"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:18:38.123604 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.123572    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01b61a33-d6ea-459e-af61-acb0728cc881-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "01b61a33-d6ea-459e-af61-acb0728cc881" (UID: "01b61a33-d6ea-459e-af61-acb0728cc881"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:18:38.123716 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.123613    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b61a33-d6ea-459e-af61-acb0728cc881-kube-api-access-2l44t" (OuterVolumeSpecName: "kube-api-access-2l44t") pod "01b61a33-d6ea-459e-af61-acb0728cc881" (UID: "01b61a33-d6ea-459e-af61-acb0728cc881"). InnerVolumeSpecName "kube-api-access-2l44t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:18:38.148798 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.148762    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b61a33-d6ea-459e-af61-acb0728cc881-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "01b61a33-d6ea-459e-af61-acb0728cc881" (UID: "01b61a33-d6ea-459e-af61-acb0728cc881"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:18:38.222634 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.221973    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01b61a33-d6ea-459e-af61-acb0728cc881-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:18:38.222634 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.222003    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01b61a33-d6ea-459e-af61-acb0728cc881-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:18:38.222634 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.222016    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l44t\" (UniqueName: \"kubernetes.io/projected/01b61a33-d6ea-459e-af61-acb0728cc881-kube-api-access-2l44t\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:18:38.222634 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.222029    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01b61a33-d6ea-459e-af61-acb0728cc881-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:18:38.222634 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.222041    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01b61a33-d6ea-459e-af61-acb0728cc881-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:18:38.259006 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.258971    2559 generic.go:356] "Generic (PLEG): container finished" podID="01b61a33-d6ea-459e-af61-acb0728cc881" containerID="7367eb8ccf11fa790d5fc14e4abd964917df169b1cc86a814a5e06e667a53c7b" exitCode=0
Jul 31 15:18:38.259165 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.259032    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"
Jul 31 15:18:38.259165 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.259044    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9" event={"ID":"01b61a33-d6ea-459e-af61-acb0728cc881","Type":"ContainerDied","Data":"7367eb8ccf11fa790d5fc14e4abd964917df169b1cc86a814a5e06e667a53c7b"}
Jul 31 15:18:38.259165 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.259104    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9" event={"ID":"01b61a33-d6ea-459e-af61-acb0728cc881","Type":"ContainerDied","Data":"d93589b1a4786d2360115432d3f8c57c615dc568f7e032b821f0b974d261042d"}
Jul 31 15:18:38.259165 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.259133    2559 scope.go:117] "RemoveContainer" containerID="11d8dbf80774b3433301ef52f046cc817cf1f014341ebf5542f8cdf1ef06f054"
Jul 31 15:18:38.268323 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.268302    2559 scope.go:117] "RemoveContainer" containerID="7367eb8ccf11fa790d5fc14e4abd964917df169b1cc86a814a5e06e667a53c7b"
Jul 31 15:18:38.276538 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.276514    2559 scope.go:117] "RemoveContainer" containerID="9cc896c27b0d3fdb8c5b5405f5aa37ff5bcdd31e837db23b32e6868f7e12be62"
Jul 31 15:18:38.278773 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.278748    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"]
Jul 31 15:18:38.284714 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.284693    2559 scope.go:117] "RemoveContainer" containerID="11d8dbf80774b3433301ef52f046cc817cf1f014341ebf5542f8cdf1ef06f054"
Jul 31 15:18:38.285038 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:18:38.285013    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11d8dbf80774b3433301ef52f046cc817cf1f014341ebf5542f8cdf1ef06f054\": container with ID starting with 11d8dbf80774b3433301ef52f046cc817cf1f014341ebf5542f8cdf1ef06f054 not found: ID does not exist" containerID="11d8dbf80774b3433301ef52f046cc817cf1f014341ebf5542f8cdf1ef06f054"
Jul 31 15:18:38.285119 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.285046    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d8dbf80774b3433301ef52f046cc817cf1f014341ebf5542f8cdf1ef06f054"} err="failed to get container status \"11d8dbf80774b3433301ef52f046cc817cf1f014341ebf5542f8cdf1ef06f054\": rpc error: code = NotFound desc = could not find container \"11d8dbf80774b3433301ef52f046cc817cf1f014341ebf5542f8cdf1ef06f054\": container with ID starting with 11d8dbf80774b3433301ef52f046cc817cf1f014341ebf5542f8cdf1ef06f054 not found: ID does not exist"
Jul 31 15:18:38.285119 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.285073    2559 scope.go:117] "RemoveContainer" containerID="7367eb8ccf11fa790d5fc14e4abd964917df169b1cc86a814a5e06e667a53c7b"
Jul 31 15:18:38.285358 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:18:38.285340    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7367eb8ccf11fa790d5fc14e4abd964917df169b1cc86a814a5e06e667a53c7b\": container with ID starting with 7367eb8ccf11fa790d5fc14e4abd964917df169b1cc86a814a5e06e667a53c7b not found: ID does not exist" containerID="7367eb8ccf11fa790d5fc14e4abd964917df169b1cc86a814a5e06e667a53c7b"
Jul 31 15:18:38.285399 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.285364    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7367eb8ccf11fa790d5fc14e4abd964917df169b1cc86a814a5e06e667a53c7b"} err="failed to get container status \"7367eb8ccf11fa790d5fc14e4abd964917df169b1cc86a814a5e06e667a53c7b\": rpc error: code = NotFound desc = could not find container \"7367eb8ccf11fa790d5fc14e4abd964917df169b1cc86a814a5e06e667a53c7b\": container with ID starting with 7367eb8ccf11fa790d5fc14e4abd964917df169b1cc86a814a5e06e667a53c7b not found: ID does not exist"
Jul 31 15:18:38.285399 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.285380    2559 scope.go:117] "RemoveContainer" containerID="9cc896c27b0d3fdb8c5b5405f5aa37ff5bcdd31e837db23b32e6868f7e12be62"
Jul 31 15:18:38.285482 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.285464    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-67986cc9f9-9z4m9"]
Jul 31 15:18:38.285620 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:18:38.285606    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cc896c27b0d3fdb8c5b5405f5aa37ff5bcdd31e837db23b32e6868f7e12be62\": container with ID starting with 9cc896c27b0d3fdb8c5b5405f5aa37ff5bcdd31e837db23b32e6868f7e12be62 not found: ID does not exist" containerID="9cc896c27b0d3fdb8c5b5405f5aa37ff5bcdd31e837db23b32e6868f7e12be62"
Jul 31 15:18:38.285664 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.285622    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cc896c27b0d3fdb8c5b5405f5aa37ff5bcdd31e837db23b32e6868f7e12be62"} err="failed to get container status \"9cc896c27b0d3fdb8c5b5405f5aa37ff5bcdd31e837db23b32e6868f7e12be62\": rpc error: code = NotFound desc = could not find container \"9cc896c27b0d3fdb8c5b5405f5aa37ff5bcdd31e837db23b32e6868f7e12be62\": container with ID starting with 9cc896c27b0d3fdb8c5b5405f5aa37ff5bcdd31e837db23b32e6868f7e12be62 not found: ID does not exist"
Jul 31 15:18:38.987241 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.987201    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh" podUID="c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 31 15:18:38.987420 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:38.987202    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w" podUID="37e06607-0588-4830-9530-612d93b90817" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 31 15:18:40.220604 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:40.220560    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01b61a33-d6ea-459e-af61-acb0728cc881" path="/var/lib/kubelet/pods/01b61a33-d6ea-459e-af61-acb0728cc881/volumes"
Jul 31 15:18:41.216992 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:41.216938    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w" podUID="37e06607-0588-4830-9530-612d93b90817" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 31 15:18:42.062899 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.062861    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"]
Jul 31 15:18:42.063390 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.063264    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2e6aa46-a3e9-4993-be7d-00fdec10e75a" containerName="kserve-container"
Jul 31 15:18:42.063390 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.063281    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2e6aa46-a3e9-4993-be7d-00fdec10e75a" containerName="kserve-container"
Jul 31 15:18:42.063390 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.063295    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2e6aa46-a3e9-4993-be7d-00fdec10e75a" containerName="kube-rbac-proxy"
Jul 31 15:18:42.063390 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.063303    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2e6aa46-a3e9-4993-be7d-00fdec10e75a" containerName="kube-rbac-proxy"
Jul 31 15:18:42.063390 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.063321    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b61a33-d6ea-459e-af61-acb0728cc881" containerName="kube-rbac-proxy"
Jul 31 15:18:42.063390 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.063329    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b61a33-d6ea-459e-af61-acb0728cc881" containerName="kube-rbac-proxy"
Jul 31 15:18:42.063390 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.063356    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b61a33-d6ea-459e-af61-acb0728cc881" containerName="kserve-container"
Jul 31 15:18:42.063390 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.063363    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b61a33-d6ea-459e-af61-acb0728cc881" containerName="kserve-container"
Jul 31 15:18:42.063390 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.063374    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2e6aa46-a3e9-4993-be7d-00fdec10e75a" containerName="storage-initializer"
Jul 31 15:18:42.063390 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.063381    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2e6aa46-a3e9-4993-be7d-00fdec10e75a" containerName="storage-initializer"
Jul 31 15:18:42.063390 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.063391    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b61a33-d6ea-459e-af61-acb0728cc881" containerName="storage-initializer"
Jul 31 15:18:42.063967 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.063399    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b61a33-d6ea-459e-af61-acb0728cc881" containerName="storage-initializer"
Jul 31 15:18:42.063967 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.063474    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="01b61a33-d6ea-459e-af61-acb0728cc881" containerName="kserve-container"
Jul 31 15:18:42.063967 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.063489    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2e6aa46-a3e9-4993-be7d-00fdec10e75a" containerName="kserve-container"
Jul 31 15:18:42.063967 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.063497    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2e6aa46-a3e9-4993-be7d-00fdec10e75a" containerName="kube-rbac-proxy"
Jul 31 15:18:42.063967 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.063506    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="01b61a33-d6ea-459e-af61-acb0728cc881" containerName="kube-rbac-proxy"
Jul 31 15:18:42.068254 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.068232    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"
Jul 31 15:18:42.071122 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.071098    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Jul 31 15:18:42.071252 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.071108    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 31 15:18:42.082572 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.082549    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"]
Jul 31 15:18:42.154048 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.154006    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52680717-9088-49bb-95c9-d55a0a38bd59-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2\" (UID: \"52680717-9088-49bb-95c9-d55a0a38bd59\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"
Jul 31 15:18:42.154048 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.154043    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52680717-9088-49bb-95c9-d55a0a38bd59-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2\" (UID: \"52680717-9088-49bb-95c9-d55a0a38bd59\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"
Jul 31 15:18:42.154309 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.154093    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngzqt\" (UniqueName: \"kubernetes.io/projected/52680717-9088-49bb-95c9-d55a0a38bd59-kube-api-access-ngzqt\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2\" (UID: \"52680717-9088-49bb-95c9-d55a0a38bd59\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"
Jul 31 15:18:42.154309 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.154131    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52680717-9088-49bb-95c9-d55a0a38bd59-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2\" (UID: \"52680717-9088-49bb-95c9-d55a0a38bd59\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"
Jul 31 15:18:42.154309 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.154180    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52680717-9088-49bb-95c9-d55a0a38bd59-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2\" (UID: \"52680717-9088-49bb-95c9-d55a0a38bd59\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"
Jul 31 15:18:42.255572 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.255530    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52680717-9088-49bb-95c9-d55a0a38bd59-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2\" (UID: \"52680717-9088-49bb-95c9-d55a0a38bd59\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"
Jul 31 15:18:42.255572 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.255571    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52680717-9088-49bb-95c9-d55a0a38bd59-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2\" (UID: \"52680717-9088-49bb-95c9-d55a0a38bd59\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"
Jul 31 15:18:42.255866 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.255602    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngzqt\" (UniqueName: \"kubernetes.io/projected/52680717-9088-49bb-95c9-d55a0a38bd59-kube-api-access-ngzqt\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2\" (UID: \"52680717-9088-49bb-95c9-d55a0a38bd59\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"
Jul 31 15:18:42.255866 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.255621    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52680717-9088-49bb-95c9-d55a0a38bd59-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2\" (UID: \"52680717-9088-49bb-95c9-d55a0a38bd59\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"
Jul 31 15:18:42.255866 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.255661    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52680717-9088-49bb-95c9-d55a0a38bd59-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2\" (UID: \"52680717-9088-49bb-95c9-d55a0a38bd59\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"
Jul 31 15:18:42.256174 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.256151    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52680717-9088-49bb-95c9-d55a0a38bd59-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2\" (UID: \"52680717-9088-49bb-95c9-d55a0a38bd59\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"
Jul 31 15:18:42.256253 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.256240    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52680717-9088-49bb-95c9-d55a0a38bd59-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2\" (UID: \"52680717-9088-49bb-95c9-d55a0a38bd59\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"
Jul 31 15:18:42.256338 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.256319    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52680717-9088-49bb-95c9-d55a0a38bd59-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2\" (UID: \"52680717-9088-49bb-95c9-d55a0a38bd59\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"
Jul 31 15:18:42.258350 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.258331    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52680717-9088-49bb-95c9-d55a0a38bd59-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2\" (UID: \"52680717-9088-49bb-95c9-d55a0a38bd59\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"
Jul 31 15:18:42.265150 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.265123    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngzqt\" (UniqueName: \"kubernetes.io/projected/52680717-9088-49bb-95c9-d55a0a38bd59-kube-api-access-ngzqt\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2\" (UID: \"52680717-9088-49bb-95c9-d55a0a38bd59\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"
Jul 31 15:18:42.380111 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.380070    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"
Jul 31 15:18:42.513849 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:42.513791    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"]
Jul 31 15:18:42.515923 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:18:42.515895    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52680717_9088_49bb_95c9_d55a0a38bd59.slice/crio-81e5558cf4ed0f8d9a1f21c92446ac47ddb10cf000c319d7ac54eeefb2f15827 WatchSource:0}: Error finding container 81e5558cf4ed0f8d9a1f21c92446ac47ddb10cf000c319d7ac54eeefb2f15827: Status 404 returned error can't find the container with id 81e5558cf4ed0f8d9a1f21c92446ac47ddb10cf000c319d7ac54eeefb2f15827
Jul 31 15:18:43.281959 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:43.281927    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2" event={"ID":"52680717-9088-49bb-95c9-d55a0a38bd59","Type":"ContainerStarted","Data":"58f4fab1a41e475e91a00bdaf189449c825748b52d4a8cfb73cacece36f2bebd"}
Jul 31 15:18:43.282326 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:43.281985    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2" event={"ID":"52680717-9088-49bb-95c9-d55a0a38bd59","Type":"ContainerStarted","Data":"81e5558cf4ed0f8d9a1f21c92446ac47ddb10cf000c319d7ac54eeefb2f15827"}
Jul 31 15:18:47.216546 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:47.216505    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh" podUID="c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 31 15:18:47.300476 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:47.300436    2559 generic.go:356] "Generic (PLEG): container finished" podID="52680717-9088-49bb-95c9-d55a0a38bd59" containerID="58f4fab1a41e475e91a00bdaf189449c825748b52d4a8cfb73cacece36f2bebd" exitCode=0
Jul 31 15:18:47.300476 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:47.300488    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2" event={"ID":"52680717-9088-49bb-95c9-d55a0a38bd59","Type":"ContainerDied","Data":"58f4fab1a41e475e91a00bdaf189449c825748b52d4a8cfb73cacece36f2bebd"}
Jul 31 15:18:48.306710 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:48.306675    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2" event={"ID":"52680717-9088-49bb-95c9-d55a0a38bd59","Type":"ContainerStarted","Data":"76fb72521cf26880273e72734546737e55eb329cad7d85261d27bddb255fe9c1"}
Jul 31 15:18:48.307156 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:48.306761    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2" event={"ID":"52680717-9088-49bb-95c9-d55a0a38bd59","Type":"ContainerStarted","Data":"fb16c5b5e38c108b9093cde4e47db987a8bdd3fe065f7efe4ed9e9d55fac0563"}
Jul 31 15:18:48.307156 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:48.307048    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"
Jul 31 15:18:48.307156 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:48.307105    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"
Jul 31 15:18:48.334076 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:48.334024    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2" podStartSLOduration=2.610866636 podStartE2EDuration="6.334003904s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.723137268s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:18:48.332676337 +0000 UTC m=+1824.644233959" watchObservedRunningTime="2026-07-31 15:18:48.334003904 +0000 UTC m=+1824.645561526"
Jul 31 15:18:51.217932 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:51.217902    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"
Jul 31 15:18:54.316081 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:54.316051    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"
Jul 31 15:18:57.216903 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:18:57.216869    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"
Jul 31 15:19:06.006534 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:06.006052    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"]
Jul 31 15:19:06.006534 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:06.006382    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w" podUID="37e06607-0588-4830-9530-612d93b90817" containerName="kserve-container" containerID="cri-o://0e7ed24e138cceb6da591fda60cda751824698e2d64bdf5187556a39a632ef18" gracePeriod=30
Jul 31 15:19:06.007193 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:06.006593    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w" podUID="37e06607-0588-4830-9530-612d93b90817" containerName="kube-rbac-proxy" containerID="cri-o://7213ee272490ab50c492d22bd076b4a1c2cfbdefe15c32e840abd5e493d07fe5" gracePeriod=30
Jul 31 15:19:06.011472 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:06.011442    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"]
Jul 31 15:19:06.011685 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:06.011663    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh" podUID="c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" containerName="kserve-container" containerID="cri-o://e5e374d731a423f87cd2f4ede655a22a9675b6ece9e2c007ce5294906a625150" gracePeriod=30
Jul 31 15:19:06.011836 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:06.011789    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh" podUID="c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" containerName="kube-rbac-proxy" containerID="cri-o://09b46f76e99a6ee0cdff94f90b7ed0f29e8e2a2b77f0d9571a105a50604be37f" gracePeriod=30
Jul 31 15:19:06.381701 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:06.381668    2559 generic.go:356] "Generic (PLEG): container finished" podID="37e06607-0588-4830-9530-612d93b90817" containerID="7213ee272490ab50c492d22bd076b4a1c2cfbdefe15c32e840abd5e493d07fe5" exitCode=2
Jul 31 15:19:06.381897 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:06.381755    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w" event={"ID":"37e06607-0588-4830-9530-612d93b90817","Type":"ContainerDied","Data":"7213ee272490ab50c492d22bd076b4a1c2cfbdefe15c32e840abd5e493d07fe5"}
Jul 31 15:19:06.383835 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:06.383817    2559 generic.go:356] "Generic (PLEG): container finished" podID="c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" containerID="09b46f76e99a6ee0cdff94f90b7ed0f29e8e2a2b77f0d9571a105a50604be37f" exitCode=2
Jul 31 15:19:06.383894 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:06.383860    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh" event={"ID":"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c","Type":"ContainerDied","Data":"09b46f76e99a6ee0cdff94f90b7ed0f29e8e2a2b77f0d9571a105a50604be37f"}
Jul 31 15:19:07.216558 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:07.216519    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh" podUID="c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 31 15:19:08.981973 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:08.981926    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w" podUID="37e06607-0588-4830-9530-612d93b90817" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Jul 31 15:19:08.982388 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:08.981930    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh" podUID="c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 31 15:19:11.054118 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.054086    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"
Jul 31 15:19:11.191989 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.191894    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"]
Jul 31 15:19:11.192276 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.192257    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" containerName="kserve-container"
Jul 31 15:19:11.192398 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.192279    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" containerName="kserve-container"
Jul 31 15:19:11.192398 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.192293    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" containerName="kube-rbac-proxy"
Jul 31 15:19:11.192398 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.192301    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" containerName="kube-rbac-proxy"
Jul 31 15:19:11.192398 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.192319    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" containerName="storage-initializer"
Jul 31 15:19:11.192398 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.192327    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" containerName="storage-initializer"
Jul 31 15:19:11.192699 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.192422    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" containerName="kube-rbac-proxy"
Jul 31 15:19:11.192699 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.192436    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" containerName="kserve-container"
Jul 31 15:19:11.194541 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.194519    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-proxy-tls\") pod \"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c\" (UID: \"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c\") "
Jul 31 15:19:11.194650 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.194584    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-cabundle-cert\") pod \"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c\" (UID: \"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c\") "
Jul 31 15:19:11.194650 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.194608    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vp5c\" (UniqueName: \"kubernetes.io/projected/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-kube-api-access-9vp5c\") pod \"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c\" (UID: \"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c\") "
Jul 31 15:19:11.194650 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.194642    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-kserve-provision-location\") pod \"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c\" (UID: \"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c\") "
Jul 31 15:19:11.194852 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.194693    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-kube-rbac-proxy-sar-config\") pod \"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c\" (UID: \"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c\") "
Jul 31 15:19:11.195007 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.194979    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" (UID: "c8930d19-1ea2-4fd8-88b0-77edd73d4d6c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:19:11.195131 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.195004    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" (UID: "c8930d19-1ea2-4fd8-88b0-77edd73d4d6c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:19:11.195131 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.195043    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" (UID: "c8930d19-1ea2-4fd8-88b0-77edd73d4d6c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:19:11.195769 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.195750    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"
Jul 31 15:19:11.198148 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.198126    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Jul 31 15:19:11.198308 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.198291    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 31 15:19:11.210214 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.210008    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"]
Jul 31 15:19:11.216875 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.216841    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w" podUID="37e06607-0588-4830-9530-612d93b90817" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 31 15:19:11.233854 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.233825    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"]
Jul 31 15:19:11.237539 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.237519    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"
Jul 31 15:19:11.240078 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.240048    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 31 15:19:11.247763 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.247715    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"]
Jul 31 15:19:11.264221 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.264181    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" (UID: "c8930d19-1ea2-4fd8-88b0-77edd73d4d6c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:19:11.264389 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.264241    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-kube-api-access-9vp5c" (OuterVolumeSpecName: "kube-api-access-9vp5c") pod "c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" (UID: "c8930d19-1ea2-4fd8-88b0-77edd73d4d6c"). InnerVolumeSpecName "kube-api-access-9vp5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:19:11.295251 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.295218    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk6t2\" (UniqueName: \"kubernetes.io/projected/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-kube-api-access-bk6t2\") pod \"isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9\" (UID: \"073437ef-93ac-4ef0-85e4-8e8c495c8d7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"
Jul 31 15:19:11.295452 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.295281    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9\" (UID: \"073437ef-93ac-4ef0-85e4-8e8c495c8d7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"
Jul 31 15:19:11.295452 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.295357    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-proxy-tls\") pod \"isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9\" (UID: \"073437ef-93ac-4ef0-85e4-8e8c495c8d7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"
Jul 31 15:19:11.295452 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.295390    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9\" (UID: \"073437ef-93ac-4ef0-85e4-8e8c495c8d7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"
Jul 31 15:19:11.295452 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.295411    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-cabundle-cert\") pod \"isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9\" (UID: \"073437ef-93ac-4ef0-85e4-8e8c495c8d7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"
Jul 31 15:19:11.295630 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.295469    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:19:11.295630 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.295480    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vp5c\" (UniqueName: \"kubernetes.io/projected/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-kube-api-access-9vp5c\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:19:11.295630 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.295489    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:19:11.295630 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.295504    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:19:11.295630 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.295522    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:19:11.396248 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.396215    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9\" (UID: \"073437ef-93ac-4ef0-85e4-8e8c495c8d7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"
Jul 31 15:19:11.396410 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.396258    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/872b0fc9-0302-4428-8784-a0a1e443db3f-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-sk628\" (UID: \"872b0fc9-0302-4428-8784-a0a1e443db3f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"
Jul 31 15:19:11.396410 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.396298    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-proxy-tls\") pod \"isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9\" (UID: \"073437ef-93ac-4ef0-85e4-8e8c495c8d7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"
Jul 31 15:19:11.396410 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.396315    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9\" (UID: \"073437ef-93ac-4ef0-85e4-8e8c495c8d7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"
Jul 31 15:19:11.396410 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.396332    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-cabundle-cert\") pod \"isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9\" (UID: \"073437ef-93ac-4ef0-85e4-8e8c495c8d7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"
Jul 31 15:19:11.396410 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.396358    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/872b0fc9-0302-4428-8784-a0a1e443db3f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-sk628\" (UID: \"872b0fc9-0302-4428-8784-a0a1e443db3f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"
Jul 31 15:19:11.396410 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.396380    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfnxv\" (UniqueName: \"kubernetes.io/projected/872b0fc9-0302-4428-8784-a0a1e443db3f-kube-api-access-zfnxv\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-sk628\" (UID: \"872b0fc9-0302-4428-8784-a0a1e443db3f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"
Jul 31 15:19:11.396410 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.396398    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk6t2\" (UniqueName: \"kubernetes.io/projected/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-kube-api-access-bk6t2\") pod \"isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9\" (UID: \"073437ef-93ac-4ef0-85e4-8e8c495c8d7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"
Jul 31 15:19:11.396410 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.396414    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/872b0fc9-0302-4428-8784-a0a1e443db3f-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-sk628\" (UID: \"872b0fc9-0302-4428-8784-a0a1e443db3f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"
Jul 31 15:19:11.396712 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.396441    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/872b0fc9-0302-4428-8784-a0a1e443db3f-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-sk628\" (UID: \"872b0fc9-0302-4428-8784-a0a1e443db3f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"
Jul 31 15:19:11.396769 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.396747    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9\" (UID: \"073437ef-93ac-4ef0-85e4-8e8c495c8d7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"
Jul 31 15:19:11.396987 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.396970    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9\" (UID: \"073437ef-93ac-4ef0-85e4-8e8c495c8d7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"
Jul 31 15:19:11.397066 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.397049    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-cabundle-cert\") pod \"isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9\" (UID: \"073437ef-93ac-4ef0-85e4-8e8c495c8d7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"
Jul 31 15:19:11.399055 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.399034    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-proxy-tls\") pod \"isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9\" (UID: \"073437ef-93ac-4ef0-85e4-8e8c495c8d7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"
Jul 31 15:19:11.405482 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.405457    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk6t2\" (UniqueName: \"kubernetes.io/projected/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-kube-api-access-bk6t2\") pod \"isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9\" (UID: \"073437ef-93ac-4ef0-85e4-8e8c495c8d7d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"
Jul 31 15:19:11.406176 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.406150    2559 generic.go:356] "Generic (PLEG): container finished" podID="37e06607-0588-4830-9530-612d93b90817" containerID="0e7ed24e138cceb6da591fda60cda751824698e2d64bdf5187556a39a632ef18" exitCode=0
Jul 31 15:19:11.406276 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.406211    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w" event={"ID":"37e06607-0588-4830-9530-612d93b90817","Type":"ContainerDied","Data":"0e7ed24e138cceb6da591fda60cda751824698e2d64bdf5187556a39a632ef18"}
Jul 31 15:19:11.411622 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.411593    2559 generic.go:356] "Generic (PLEG): container finished" podID="c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" containerID="e5e374d731a423f87cd2f4ede655a22a9675b6ece9e2c007ce5294906a625150" exitCode=0
Jul 31 15:19:11.411791 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.411664    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"
Jul 31 15:19:11.411791 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.411669    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh" event={"ID":"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c","Type":"ContainerDied","Data":"e5e374d731a423f87cd2f4ede655a22a9675b6ece9e2c007ce5294906a625150"}
Jul 31 15:19:11.411791 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.411746    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh" event={"ID":"c8930d19-1ea2-4fd8-88b0-77edd73d4d6c","Type":"ContainerDied","Data":"00b28e2994610cd77f8c7b00f97e2ec2bf5d8512edba7cdc06a1a2e8e5ffe0e4"}
Jul 31 15:19:11.411791 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.411783    2559 scope.go:117] "RemoveContainer" containerID="09b46f76e99a6ee0cdff94f90b7ed0f29e8e2a2b77f0d9571a105a50604be37f"
Jul 31 15:19:11.420950 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.420864    2559 scope.go:117] "RemoveContainer" containerID="e5e374d731a423f87cd2f4ede655a22a9675b6ece9e2c007ce5294906a625150"
Jul 31 15:19:11.428501 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.428485    2559 scope.go:117] "RemoveContainer" containerID="037bb01f5cc01be30a363f322ceda0aadcea5fb267790d42b583a04fa7ec1edf"
Jul 31 15:19:11.436378 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.436354    2559 scope.go:117] "RemoveContainer" containerID="09b46f76e99a6ee0cdff94f90b7ed0f29e8e2a2b77f0d9571a105a50604be37f"
Jul 31 15:19:11.436695 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:19:11.436674    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09b46f76e99a6ee0cdff94f90b7ed0f29e8e2a2b77f0d9571a105a50604be37f\": container with ID starting with 09b46f76e99a6ee0cdff94f90b7ed0f29e8e2a2b77f0d9571a105a50604be37f not found: ID does not exist" containerID="09b46f76e99a6ee0cdff94f90b7ed0f29e8e2a2b77f0d9571a105a50604be37f"
Jul 31 15:19:11.436815 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.436705    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09b46f76e99a6ee0cdff94f90b7ed0f29e8e2a2b77f0d9571a105a50604be37f"} err="failed to get container status \"09b46f76e99a6ee0cdff94f90b7ed0f29e8e2a2b77f0d9571a105a50604be37f\": rpc error: code = NotFound desc = could not find container \"09b46f76e99a6ee0cdff94f90b7ed0f29e8e2a2b77f0d9571a105a50604be37f\": container with ID starting with 09b46f76e99a6ee0cdff94f90b7ed0f29e8e2a2b77f0d9571a105a50604be37f not found: ID does not exist"
Jul 31 15:19:11.436815 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.436747    2559 scope.go:117] "RemoveContainer" containerID="e5e374d731a423f87cd2f4ede655a22a9675b6ece9e2c007ce5294906a625150"
Jul 31 15:19:11.437018 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:19:11.437000    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5e374d731a423f87cd2f4ede655a22a9675b6ece9e2c007ce5294906a625150\": container with ID starting with e5e374d731a423f87cd2f4ede655a22a9675b6ece9e2c007ce5294906a625150 not found: ID does not exist" containerID="e5e374d731a423f87cd2f4ede655a22a9675b6ece9e2c007ce5294906a625150"
Jul 31 15:19:11.437073 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.437023    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5e374d731a423f87cd2f4ede655a22a9675b6ece9e2c007ce5294906a625150"} err="failed to get container status \"e5e374d731a423f87cd2f4ede655a22a9675b6ece9e2c007ce5294906a625150\": rpc error: code = NotFound desc = could not find container \"e5e374d731a423f87cd2f4ede655a22a9675b6ece9e2c007ce5294906a625150\": container with ID starting with e5e374d731a423f87cd2f4ede655a22a9675b6ece9e2c007ce5294906a625150 not found: ID does not exist"
Jul 31 15:19:11.437073 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.437039    2559 scope.go:117] "RemoveContainer" containerID="037bb01f5cc01be30a363f322ceda0aadcea5fb267790d42b583a04fa7ec1edf"
Jul 31 15:19:11.437289 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:19:11.437273    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"037bb01f5cc01be30a363f322ceda0aadcea5fb267790d42b583a04fa7ec1edf\": container with ID starting with 037bb01f5cc01be30a363f322ceda0aadcea5fb267790d42b583a04fa7ec1edf not found: ID does not exist" containerID="037bb01f5cc01be30a363f322ceda0aadcea5fb267790d42b583a04fa7ec1edf"
Jul 31 15:19:11.437352 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.437292    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"037bb01f5cc01be30a363f322ceda0aadcea5fb267790d42b583a04fa7ec1edf"} err="failed to get container status \"037bb01f5cc01be30a363f322ceda0aadcea5fb267790d42b583a04fa7ec1edf\": rpc error: code = NotFound desc = could not find container \"037bb01f5cc01be30a363f322ceda0aadcea5fb267790d42b583a04fa7ec1edf\": container with ID starting with 037bb01f5cc01be30a363f322ceda0aadcea5fb267790d42b583a04fa7ec1edf not found: ID does not exist"
Jul 31 15:19:11.438006 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.437986    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"]
Jul 31 15:19:11.442101 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.442050    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5cd5878b66-ptnsh"]
Jul 31 15:19:11.454835 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.454813    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"
Jul 31 15:19:11.497600 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.497554    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/872b0fc9-0302-4428-8784-a0a1e443db3f-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-sk628\" (UID: \"872b0fc9-0302-4428-8784-a0a1e443db3f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"
Jul 31 15:19:11.497600 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.497609    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/872b0fc9-0302-4428-8784-a0a1e443db3f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-sk628\" (UID: \"872b0fc9-0302-4428-8784-a0a1e443db3f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"
Jul 31 15:19:11.497896 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.497630    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfnxv\" (UniqueName: \"kubernetes.io/projected/872b0fc9-0302-4428-8784-a0a1e443db3f-kube-api-access-zfnxv\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-sk628\" (UID: \"872b0fc9-0302-4428-8784-a0a1e443db3f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"
Jul 31 15:19:11.497896 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.497648    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/872b0fc9-0302-4428-8784-a0a1e443db3f-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-sk628\" (UID: \"872b0fc9-0302-4428-8784-a0a1e443db3f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"
Jul 31 15:19:11.497896 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.497675    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/872b0fc9-0302-4428-8784-a0a1e443db3f-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-sk628\" (UID: \"872b0fc9-0302-4428-8784-a0a1e443db3f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"
Jul 31 15:19:11.498155 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.498126    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/872b0fc9-0302-4428-8784-a0a1e443db3f-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-sk628\" (UID: \"872b0fc9-0302-4428-8784-a0a1e443db3f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"
Jul 31 15:19:11.498297 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.498282    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/872b0fc9-0302-4428-8784-a0a1e443db3f-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-sk628\" (UID: \"872b0fc9-0302-4428-8784-a0a1e443db3f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"
Jul 31 15:19:11.498410 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.498391    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/872b0fc9-0302-4428-8784-a0a1e443db3f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-sk628\" (UID: \"872b0fc9-0302-4428-8784-a0a1e443db3f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"
Jul 31 15:19:11.500287 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.500268    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/872b0fc9-0302-4428-8784-a0a1e443db3f-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-sk628\" (UID: \"872b0fc9-0302-4428-8784-a0a1e443db3f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"
Jul 31 15:19:11.506755 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.506711    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfnxv\" (UniqueName: \"kubernetes.io/projected/872b0fc9-0302-4428-8784-a0a1e443db3f-kube-api-access-zfnxv\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-sk628\" (UID: \"872b0fc9-0302-4428-8784-a0a1e443db3f\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"
Jul 31 15:19:11.516693 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.516667    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"
Jul 31 15:19:11.548100 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.548062    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"
Jul 31 15:19:11.598404 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.598364    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crxzz\" (UniqueName: \"kubernetes.io/projected/37e06607-0588-4830-9530-612d93b90817-kube-api-access-crxzz\") pod \"37e06607-0588-4830-9530-612d93b90817\" (UID: \"37e06607-0588-4830-9530-612d93b90817\") "
Jul 31 15:19:11.598571 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.598415    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37e06607-0588-4830-9530-612d93b90817-kserve-provision-location\") pod \"37e06607-0588-4830-9530-612d93b90817\" (UID: \"37e06607-0588-4830-9530-612d93b90817\") "
Jul 31 15:19:11.598571 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.598444    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37e06607-0588-4830-9530-612d93b90817-proxy-tls\") pod \"37e06607-0588-4830-9530-612d93b90817\" (UID: \"37e06607-0588-4830-9530-612d93b90817\") "
Jul 31 15:19:11.598571 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.598531    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37e06607-0588-4830-9530-612d93b90817-cabundle-cert\") pod \"37e06607-0588-4830-9530-612d93b90817\" (UID: \"37e06607-0588-4830-9530-612d93b90817\") "
Jul 31 15:19:11.598772 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.598576    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37e06607-0588-4830-9530-612d93b90817-kube-rbac-proxy-sar-config\") pod \"37e06607-0588-4830-9530-612d93b90817\" (UID: \"37e06607-0588-4830-9530-612d93b90817\") "
Jul 31 15:19:11.598967 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.598936    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37e06607-0588-4830-9530-612d93b90817-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "37e06607-0588-4830-9530-612d93b90817" (UID: "37e06607-0588-4830-9530-612d93b90817"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:19:11.599075 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.599052    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37e06607-0588-4830-9530-612d93b90817-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "37e06607-0588-4830-9530-612d93b90817" (UID: "37e06607-0588-4830-9530-612d93b90817"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:19:11.599138 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.599090    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37e06607-0588-4830-9530-612d93b90817-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "37e06607-0588-4830-9530-612d93b90817" (UID: "37e06607-0588-4830-9530-612d93b90817"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:19:11.601155 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.601102    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37e06607-0588-4830-9530-612d93b90817-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "37e06607-0588-4830-9530-612d93b90817" (UID: "37e06607-0588-4830-9530-612d93b90817"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:19:11.601314 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.601280    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37e06607-0588-4830-9530-612d93b90817-kube-api-access-crxzz" (OuterVolumeSpecName: "kube-api-access-crxzz") pod "37e06607-0588-4830-9530-612d93b90817" (UID: "37e06607-0588-4830-9530-612d93b90817"). InnerVolumeSpecName "kube-api-access-crxzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:19:11.663094 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.663065    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"]
Jul 31 15:19:11.666353 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:19:11.666317    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod073437ef_93ac_4ef0_85e4_8e8c495c8d7d.slice/crio-c3194382b4040d2002240c4fd21ef0858cb3b359cbc53642375f2f0e871b0b9d WatchSource:0}: Error finding container c3194382b4040d2002240c4fd21ef0858cb3b359cbc53642375f2f0e871b0b9d: Status 404 returned error can't find the container with id c3194382b4040d2002240c4fd21ef0858cb3b359cbc53642375f2f0e871b0b9d
Jul 31 15:19:11.697294 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.697266    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"]
Jul 31 15:19:11.699417 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.699394    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37e06607-0588-4830-9530-612d93b90817-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:19:11.699524 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.699418    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37e06607-0588-4830-9530-612d93b90817-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:19:11.699524 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.699433    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crxzz\" (UniqueName: \"kubernetes.io/projected/37e06607-0588-4830-9530-612d93b90817-kube-api-access-crxzz\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:19:11.699524 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.699446    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37e06607-0588-4830-9530-612d93b90817-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:19:11.699524 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:11.699458    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37e06607-0588-4830-9530-612d93b90817-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:19:11.699524 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:19:11.699467    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872b0fc9_0302_4428_8784_a0a1e443db3f.slice/crio-4fd652c4427d86d0022075f42327eb53257e2e9e2482f7fbfb44c456087df2a9 WatchSource:0}: Error finding container 4fd652c4427d86d0022075f42327eb53257e2e9e2482f7fbfb44c456087df2a9: Status 404 returned error can't find the container with id 4fd652c4427d86d0022075f42327eb53257e2e9e2482f7fbfb44c456087df2a9
Jul 31 15:19:12.222368 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:12.222333    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8930d19-1ea2-4fd8-88b0-77edd73d4d6c" path="/var/lib/kubelet/pods/c8930d19-1ea2-4fd8-88b0-77edd73d4d6c/volumes"
Jul 31 15:19:12.416845 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:12.416805    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9" event={"ID":"073437ef-93ac-4ef0-85e4-8e8c495c8d7d","Type":"ContainerStarted","Data":"9bf67e069a73d829e02fcb004623c3c9418660959cbff6e16fe09589f79cbaf4"}
Jul 31 15:19:12.417015 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:12.416883    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9" event={"ID":"073437ef-93ac-4ef0-85e4-8e8c495c8d7d","Type":"ContainerStarted","Data":"c3194382b4040d2002240c4fd21ef0858cb3b359cbc53642375f2f0e871b0b9d"}
Jul 31 15:19:12.418749 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:12.418704    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"
Jul 31 15:19:12.418749 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:12.418725    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w" event={"ID":"37e06607-0588-4830-9530-612d93b90817","Type":"ContainerDied","Data":"ed43e237e5b1547cf1bb8ee08391b4c8a9d72d48ca2b0b96b4e5da5da8721e70"}
Jul 31 15:19:12.418939 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:12.418807    2559 scope.go:117] "RemoveContainer" containerID="7213ee272490ab50c492d22bd076b4a1c2cfbdefe15c32e840abd5e493d07fe5"
Jul 31 15:19:12.421601 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:12.421577    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628" event={"ID":"872b0fc9-0302-4428-8784-a0a1e443db3f","Type":"ContainerStarted","Data":"e9888e500a4231256702617eee52c81432d9fdf932c0f93d39e0a92dbeb5707b"}
Jul 31 15:19:12.421715 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:12.421630    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628" event={"ID":"872b0fc9-0302-4428-8784-a0a1e443db3f","Type":"ContainerStarted","Data":"4fd652c4427d86d0022075f42327eb53257e2e9e2482f7fbfb44c456087df2a9"}
Jul 31 15:19:12.427694 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:12.427677    2559 scope.go:117] "RemoveContainer" containerID="0e7ed24e138cceb6da591fda60cda751824698e2d64bdf5187556a39a632ef18"
Jul 31 15:19:12.435192 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:12.435173    2559 scope.go:117] "RemoveContainer" containerID="71c9a11bffea99137f25cef8b9ecf816bb34955390b57a24932cc57ad2da355b"
Jul 31 15:19:12.454232 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:12.454204    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"]
Jul 31 15:19:12.458781 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:12.458755    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5f99649479-bnz2w"]
Jul 31 15:19:14.222568 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:14.222536    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37e06607-0588-4830-9530-612d93b90817" path="/var/lib/kubelet/pods/37e06607-0588-4830-9530-612d93b90817/volumes"
Jul 31 15:19:16.438479 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:16.438444    2559 generic.go:356] "Generic (PLEG): container finished" podID="073437ef-93ac-4ef0-85e4-8e8c495c8d7d" containerID="9bf67e069a73d829e02fcb004623c3c9418660959cbff6e16fe09589f79cbaf4" exitCode=0
Jul 31 15:19:16.438994 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:16.438523    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9" event={"ID":"073437ef-93ac-4ef0-85e4-8e8c495c8d7d","Type":"ContainerDied","Data":"9bf67e069a73d829e02fcb004623c3c9418660959cbff6e16fe09589f79cbaf4"}
Jul 31 15:19:16.440058 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:16.440030    2559 generic.go:356] "Generic (PLEG): container finished" podID="872b0fc9-0302-4428-8784-a0a1e443db3f" containerID="e9888e500a4231256702617eee52c81432d9fdf932c0f93d39e0a92dbeb5707b" exitCode=0
Jul 31 15:19:16.440182 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:16.440071    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628" event={"ID":"872b0fc9-0302-4428-8784-a0a1e443db3f","Type":"ContainerDied","Data":"e9888e500a4231256702617eee52c81432d9fdf932c0f93d39e0a92dbeb5707b"}
Jul 31 15:19:17.446943 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:17.446910    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628" event={"ID":"872b0fc9-0302-4428-8784-a0a1e443db3f","Type":"ContainerStarted","Data":"dddf5141e93cc050ef660baeb29d668f584572a5e91c19d4c4943f0bff32e0c9"}
Jul 31 15:19:17.447410 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:17.446975    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628" event={"ID":"872b0fc9-0302-4428-8784-a0a1e443db3f","Type":"ContainerStarted","Data":"5803bc37185849965f862c3a30d267556f57c348df5afa55c887e85cb047a440"}
Jul 31 15:19:17.447410 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:17.447276    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"
Jul 31 15:19:17.447410 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:17.447394    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"
Jul 31 15:19:17.449103 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:17.449061    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628" podUID="872b0fc9-0302-4428-8784-a0a1e443db3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 31 15:19:17.449222 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:17.449205    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9" event={"ID":"073437ef-93ac-4ef0-85e4-8e8c495c8d7d","Type":"ContainerStarted","Data":"cef77ccb09caa9e7f3119582cb58e44606478e309a24f8fd9ebd366fd3e8f75c"}
Jul 31 15:19:17.449311 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:17.449245    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9" event={"ID":"073437ef-93ac-4ef0-85e4-8e8c495c8d7d","Type":"ContainerStarted","Data":"a01846b294367dbf2461181aff031dbea65de5b6a552a9cfe4ba0c3cfb64af31"}
Jul 31 15:19:17.449510 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:17.449494    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"
Jul 31 15:19:17.449558 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:17.449535    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"
Jul 31 15:19:17.450439 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:17.450417    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9" podUID="073437ef-93ac-4ef0-85e4-8e8c495c8d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 31 15:19:17.467881 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:17.467832    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628" podStartSLOduration=2.595938601 podStartE2EDuration="6.467816953s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.871878352s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:19:17.465910801 +0000 UTC m=+1853.777468424" watchObservedRunningTime="2026-07-31 15:19:17.467816953 +0000 UTC m=+1853.779374576"
Jul 31 15:19:17.486051 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:17.485993    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9" podStartSLOduration=2.550333054 podStartE2EDuration="6.485976713s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.935643659s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:19:17.484225255 +0000 UTC m=+1853.795782878" watchObservedRunningTime="2026-07-31 15:19:17.485976713 +0000 UTC m=+1853.797534335"
Jul 31 15:19:18.453218 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:18.453176    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628" podUID="872b0fc9-0302-4428-8784-a0a1e443db3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 31 15:19:18.453655 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:18.453268    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9" podUID="073437ef-93ac-4ef0-85e4-8e8c495c8d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 31 15:19:23.457815 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:23.457778    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"
Jul 31 15:19:23.458328 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:23.457998    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"
Jul 31 15:19:23.458394 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:23.458364    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628" podUID="872b0fc9-0302-4428-8784-a0a1e443db3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 31 15:19:23.458698 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:23.458678    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9" podUID="073437ef-93ac-4ef0-85e4-8e8c495c8d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 31 15:19:24.465849 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:24.465797    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2" podUID="52680717-9088-49bb-95c9-d55a0a38bd59" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 31 15:19:33.458773 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:33.458706    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628" podUID="872b0fc9-0302-4428-8784-a0a1e443db3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 31 15:19:33.459157 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:33.458706    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9" podUID="073437ef-93ac-4ef0-85e4-8e8c495c8d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 31 15:19:34.318971 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:34.318939    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"
Jul 31 15:19:42.126906 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:42.126870    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"]
Jul 31 15:19:42.127302 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:42.127129    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2" podUID="52680717-9088-49bb-95c9-d55a0a38bd59" containerName="kserve-container" containerID="cri-o://fb16c5b5e38c108b9093cde4e47db987a8bdd3fe065f7efe4ed9e9d55fac0563" gracePeriod=30
Jul 31 15:19:42.127302 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:42.127186    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2" podUID="52680717-9088-49bb-95c9-d55a0a38bd59" containerName="kube-rbac-proxy" containerID="cri-o://76fb72521cf26880273e72734546737e55eb329cad7d85261d27bddb255fe9c1" gracePeriod=30
Jul 31 15:19:42.552189 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:42.552103    2559 generic.go:356] "Generic (PLEG): container finished" podID="52680717-9088-49bb-95c9-d55a0a38bd59" containerID="76fb72521cf26880273e72734546737e55eb329cad7d85261d27bddb255fe9c1" exitCode=2
Jul 31 15:19:42.552189 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:42.552174    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2" event={"ID":"52680717-9088-49bb-95c9-d55a0a38bd59","Type":"ContainerDied","Data":"76fb72521cf26880273e72734546737e55eb329cad7d85261d27bddb255fe9c1"}
Jul 31 15:19:43.459241 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:43.459201    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628" podUID="872b0fc9-0302-4428-8784-a0a1e443db3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 31 15:19:43.459696 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:43.459201    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9" podUID="073437ef-93ac-4ef0-85e4-8e8c495c8d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 31 15:19:44.311745 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:44.311685    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2" podUID="52680717-9088-49bb-95c9-d55a0a38bd59" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 31 15:19:45.357992 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:45.357892    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2" podUID="52680717-9088-49bb-95c9-d55a0a38bd59" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.37:8080/v2/models/isvc-sklearn-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 31 15:19:47.382318 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:47.382282    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"]
Jul 31 15:19:47.382718 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:47.382585    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37e06607-0588-4830-9530-612d93b90817" containerName="kube-rbac-proxy"
Jul 31 15:19:47.382718 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:47.382595    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="37e06607-0588-4830-9530-612d93b90817" containerName="kube-rbac-proxy"
Jul 31 15:19:47.382718 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:47.382603    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37e06607-0588-4830-9530-612d93b90817" containerName="storage-initializer"
Jul 31 15:19:47.382718 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:47.382608    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="37e06607-0588-4830-9530-612d93b90817" containerName="storage-initializer"
Jul 31 15:19:47.382718 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:47.382615    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37e06607-0588-4830-9530-612d93b90817" containerName="kserve-container"
Jul 31 15:19:47.382718 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:47.382621    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="37e06607-0588-4830-9530-612d93b90817" containerName="kserve-container"
Jul 31 15:19:47.382718 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:47.382673    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="37e06607-0588-4830-9530-612d93b90817" containerName="kserve-container"
Jul 31 15:19:47.382718 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:47.382681    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="37e06607-0588-4830-9530-612d93b90817" containerName="kube-rbac-proxy"
Jul 31 15:19:47.385436 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:47.385418    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"
Jul 31 15:19:47.388027 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:47.388005    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Jul 31 15:19:47.388316 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:47.388301    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 31 15:19:47.400287 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:47.400265    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"]
Jul 31 15:19:47.478011 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:47.477976    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npfxw\" (UniqueName: \"kubernetes.io/projected/cd08fe2e-e0e6-473b-a431-e03e3441d28e-kube-api-access-npfxw\") pod \"isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr\" (UID: \"cd08fe2e-e0e6-473b-a431-e03e3441d28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"
Jul 31 15:19:47.478180 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:47.478053    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd08fe2e-e0e6-473b-a431-e03e3441d28e-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr\" (UID: \"cd08fe2e-e0e6-473b-a431-e03e3441d28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"
Jul 31 15:19:47.478180 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:47.478072    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd08fe2e-e0e6-473b-a431-e03e3441d28e-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr\" (UID: \"cd08fe2e-e0e6-473b-a431-e03e3441d28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"
Jul 31 15:19:47.478180 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:47.478090    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd08fe2e-e0e6-473b-a431-e03e3441d28e-proxy-tls\") pod \"isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr\" (UID: \"cd08fe2e-e0e6-473b-a431-e03e3441d28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"
Jul 31 15:19:47.478180 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:47.478112    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd08fe2e-e0e6-473b-a431-e03e3441d28e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr\" (UID: \"cd08fe2e-e0e6-473b-a431-e03e3441d28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"
Jul 31 15:19:47.578996 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:47.578967    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npfxw\" (UniqueName: \"kubernetes.io/projected/cd08fe2e-e0e6-473b-a431-e03e3441d28e-kube-api-access-npfxw\") pod \"isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr\" (UID: \"cd08fe2e-e0e6-473b-a431-e03e3441d28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"
Jul 31 15:19:47.579185 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:47.579010    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd08fe2e-e0e6-473b-a431-e03e3441d28e-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr\" (UID: \"cd08fe2e-e0e6-473b-a431-e03e3441d28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"
Jul 31 15:19:47.579185 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:47.579029    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd08fe2e-e0e6-473b-a431-e03e3441d28e-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr\" (UID: \"cd08fe2e-e0e6-473b-a431-e03e3441d28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"
Jul 31 15:19:47.579185 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:47.579048    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd08fe2e-e0e6-473b-a431-e03e3441d28e-proxy-tls\") pod \"isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr\" (UID: \"cd08fe2e-e0e6-473b-a431-e03e3441d28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"
Jul 31 15:19:47.579185 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:47.579073    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd08fe2e-e0e6-473b-a431-e03e3441d28e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr\" (UID: \"cd08fe2e-e0e6-473b-a431-e03e3441d28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"
Jul 31 15:19:47.579398 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:19:47.579246    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
Jul 31 15:19:47.579398 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:19:47.579339    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd08fe2e-e0e6-473b-a431-e03e3441d28e-proxy-tls podName:cd08fe2e-e0e6-473b-a431-e03e3441d28e nodeName:}" failed. No retries permitted until 2026-07-31 15:19:48.079315642 +0000 UTC m=+1884.390873263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cd08fe2e-e0e6-473b-a431-e03e3441d28e-proxy-tls") pod "isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr" (UID: "cd08fe2e-e0e6-473b-a431-e03e3441d28e") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
Jul 31 15:19:47.579475 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:47.579392    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd08fe2e-e0e6-473b-a431-e03e3441d28e-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr\" (UID: \"cd08fe2e-e0e6-473b-a431-e03e3441d28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"
Jul 31 15:19:47.579683 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:47.579665    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd08fe2e-e0e6-473b-a431-e03e3441d28e-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr\" (UID: \"cd08fe2e-e0e6-473b-a431-e03e3441d28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"
Jul 31 15:19:47.579716 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:47.579683    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd08fe2e-e0e6-473b-a431-e03e3441d28e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr\" (UID: \"cd08fe2e-e0e6-473b-a431-e03e3441d28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"
Jul 31 15:19:47.590764 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:47.590721    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npfxw\" (UniqueName: \"kubernetes.io/projected/cd08fe2e-e0e6-473b-a431-e03e3441d28e-kube-api-access-npfxw\") pod \"isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr\" (UID: \"cd08fe2e-e0e6-473b-a431-e03e3441d28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"
Jul 31 15:19:48.082912 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:48.082875    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd08fe2e-e0e6-473b-a431-e03e3441d28e-proxy-tls\") pod \"isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr\" (UID: \"cd08fe2e-e0e6-473b-a431-e03e3441d28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"
Jul 31 15:19:48.085531 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:48.085503    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd08fe2e-e0e6-473b-a431-e03e3441d28e-proxy-tls\") pod \"isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr\" (UID: \"cd08fe2e-e0e6-473b-a431-e03e3441d28e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"
Jul 31 15:19:48.295567 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:48.295528    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"
Jul 31 15:19:48.437880 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:48.437685    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"]
Jul 31 15:19:48.440487 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:19:48.440461    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd08fe2e_e0e6_473b_a431_e03e3441d28e.slice/crio-16668d0ae9eb4d685f5633b82adf4e613644aee2291e86966015d0e4bc9983c7 WatchSource:0}: Error finding container 16668d0ae9eb4d685f5633b82adf4e613644aee2291e86966015d0e4bc9983c7: Status 404 returned error can't find the container with id 16668d0ae9eb4d685f5633b82adf4e613644aee2291e86966015d0e4bc9983c7
Jul 31 15:19:48.579324 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:48.579291    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr" event={"ID":"cd08fe2e-e0e6-473b-a431-e03e3441d28e","Type":"ContainerStarted","Data":"908c0b0dbd7015b9340a11f0d79cc8eb443058af33d7ba8fcaa22417a196ca1c"}
Jul 31 15:19:48.579490 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:48.579349    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr" event={"ID":"cd08fe2e-e0e6-473b-a431-e03e3441d28e","Type":"ContainerStarted","Data":"16668d0ae9eb4d685f5633b82adf4e613644aee2291e86966015d0e4bc9983c7"}
Jul 31 15:19:49.312365 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:49.312319    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2" podUID="52680717-9088-49bb-95c9-d55a0a38bd59" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 31 15:19:52.595492 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:52.595463    2559 generic.go:356] "Generic (PLEG): container finished" podID="cd08fe2e-e0e6-473b-a431-e03e3441d28e" containerID="908c0b0dbd7015b9340a11f0d79cc8eb443058af33d7ba8fcaa22417a196ca1c" exitCode=0
Jul 31 15:19:52.595896 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:52.595536    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr" event={"ID":"cd08fe2e-e0e6-473b-a431-e03e3441d28e","Type":"ContainerDied","Data":"908c0b0dbd7015b9340a11f0d79cc8eb443058af33d7ba8fcaa22417a196ca1c"}
Jul 31 15:19:53.458441 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:53.458394    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628" podUID="872b0fc9-0302-4428-8784-a0a1e443db3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 31 15:19:53.458648 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:53.458538    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9" podUID="073437ef-93ac-4ef0-85e4-8e8c495c8d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 31 15:19:53.601538 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:53.601504    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr" event={"ID":"cd08fe2e-e0e6-473b-a431-e03e3441d28e","Type":"ContainerStarted","Data":"56db66c16f2353b0faec99bdeca03998682a65d707ac64e0c0d43aae17c68fa6"}
Jul 31 15:19:53.601934 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:53.601562    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr" event={"ID":"cd08fe2e-e0e6-473b-a431-e03e3441d28e","Type":"ContainerStarted","Data":"08d34f9f31844105f10ac9fe05ca6477b46044d933a9c99faf24d51def59a174"}
Jul 31 15:19:53.601934 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:53.601852    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"
Jul 31 15:19:53.602041 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:53.602015    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"
Jul 31 15:19:53.603283 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:53.603260    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr" podUID="cd08fe2e-e0e6-473b-a431-e03e3441d28e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 31 15:19:53.623706 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:53.623662    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr" podStartSLOduration=3.004275496 podStartE2EDuration="6.623649599s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.619374103s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:19:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:19:53.621353417 +0000 UTC m=+1889.932911038" watchObservedRunningTime="2026-07-31 15:19:53.623649599 +0000 UTC m=+1889.935207220"
Jul 31 15:19:54.181082 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.181039    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"
Jul 31 15:19:54.232265 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.232232    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52680717-9088-49bb-95c9-d55a0a38bd59-cabundle-cert\") pod \"52680717-9088-49bb-95c9-d55a0a38bd59\" (UID: \"52680717-9088-49bb-95c9-d55a0a38bd59\") "
Jul 31 15:19:54.232452 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.232280    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52680717-9088-49bb-95c9-d55a0a38bd59-kube-rbac-proxy-sar-config\") pod \"52680717-9088-49bb-95c9-d55a0a38bd59\" (UID: \"52680717-9088-49bb-95c9-d55a0a38bd59\") "
Jul 31 15:19:54.232452 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.232326    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52680717-9088-49bb-95c9-d55a0a38bd59-proxy-tls\") pod \"52680717-9088-49bb-95c9-d55a0a38bd59\" (UID: \"52680717-9088-49bb-95c9-d55a0a38bd59\") "
Jul 31 15:19:54.232452 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.232368    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngzqt\" (UniqueName: \"kubernetes.io/projected/52680717-9088-49bb-95c9-d55a0a38bd59-kube-api-access-ngzqt\") pod \"52680717-9088-49bb-95c9-d55a0a38bd59\" (UID: \"52680717-9088-49bb-95c9-d55a0a38bd59\") "
Jul 31 15:19:54.232452 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.232415    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52680717-9088-49bb-95c9-d55a0a38bd59-kserve-provision-location\") pod \"52680717-9088-49bb-95c9-d55a0a38bd59\" (UID: \"52680717-9088-49bb-95c9-d55a0a38bd59\") "
Jul 31 15:19:54.232643 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.232578    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52680717-9088-49bb-95c9-d55a0a38bd59-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "52680717-9088-49bb-95c9-d55a0a38bd59" (UID: "52680717-9088-49bb-95c9-d55a0a38bd59"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:19:54.232702 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.232640    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52680717-9088-49bb-95c9-d55a0a38bd59-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "52680717-9088-49bb-95c9-d55a0a38bd59" (UID: "52680717-9088-49bb-95c9-d55a0a38bd59"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:19:54.232832 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.232814    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52680717-9088-49bb-95c9-d55a0a38bd59-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:19:54.232832 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.232820    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52680717-9088-49bb-95c9-d55a0a38bd59-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "52680717-9088-49bb-95c9-d55a0a38bd59" (UID: "52680717-9088-49bb-95c9-d55a0a38bd59"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:19:54.234648 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.234626    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52680717-9088-49bb-95c9-d55a0a38bd59-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "52680717-9088-49bb-95c9-d55a0a38bd59" (UID: "52680717-9088-49bb-95c9-d55a0a38bd59"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:19:54.234703 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.234682    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52680717-9088-49bb-95c9-d55a0a38bd59-kube-api-access-ngzqt" (OuterVolumeSpecName: "kube-api-access-ngzqt") pod "52680717-9088-49bb-95c9-d55a0a38bd59" (UID: "52680717-9088-49bb-95c9-d55a0a38bd59"). InnerVolumeSpecName "kube-api-access-ngzqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:19:54.333651 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.333552    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52680717-9088-49bb-95c9-d55a0a38bd59-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:19:54.333651 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.333596    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52680717-9088-49bb-95c9-d55a0a38bd59-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:19:54.333651 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.333608    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngzqt\" (UniqueName: \"kubernetes.io/projected/52680717-9088-49bb-95c9-d55a0a38bd59-kube-api-access-ngzqt\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:19:54.333651 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.333616    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52680717-9088-49bb-95c9-d55a0a38bd59-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:19:54.611193 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.611153    2559 generic.go:356] "Generic (PLEG): container finished" podID="52680717-9088-49bb-95c9-d55a0a38bd59" containerID="fb16c5b5e38c108b9093cde4e47db987a8bdd3fe065f7efe4ed9e9d55fac0563" exitCode=0
Jul 31 15:19:54.611598 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.611229    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"
Jul 31 15:19:54.611598 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.611229    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2" event={"ID":"52680717-9088-49bb-95c9-d55a0a38bd59","Type":"ContainerDied","Data":"fb16c5b5e38c108b9093cde4e47db987a8bdd3fe065f7efe4ed9e9d55fac0563"}
Jul 31 15:19:54.611598 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.611318    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2" event={"ID":"52680717-9088-49bb-95c9-d55a0a38bd59","Type":"ContainerDied","Data":"81e5558cf4ed0f8d9a1f21c92446ac47ddb10cf000c319d7ac54eeefb2f15827"}
Jul 31 15:19:54.611598 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.611344    2559 scope.go:117] "RemoveContainer" containerID="76fb72521cf26880273e72734546737e55eb329cad7d85261d27bddb255fe9c1"
Jul 31 15:19:54.612028 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.612001    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr" podUID="cd08fe2e-e0e6-473b-a431-e03e3441d28e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 31 15:19:54.620143 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.620129    2559 scope.go:117] "RemoveContainer" containerID="fb16c5b5e38c108b9093cde4e47db987a8bdd3fe065f7efe4ed9e9d55fac0563"
Jul 31 15:19:54.627463 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.627445    2559 scope.go:117] "RemoveContainer" containerID="58f4fab1a41e475e91a00bdaf189449c825748b52d4a8cfb73cacece36f2bebd"
Jul 31 15:19:54.631461 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.631438    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"]
Jul 31 15:19:54.634840 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.634823    2559 scope.go:117] "RemoveContainer" containerID="76fb72521cf26880273e72734546737e55eb329cad7d85261d27bddb255fe9c1"
Jul 31 15:19:54.635155 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:19:54.635110    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76fb72521cf26880273e72734546737e55eb329cad7d85261d27bddb255fe9c1\": container with ID starting with 76fb72521cf26880273e72734546737e55eb329cad7d85261d27bddb255fe9c1 not found: ID does not exist" containerID="76fb72521cf26880273e72734546737e55eb329cad7d85261d27bddb255fe9c1"
Jul 31 15:19:54.635155 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.635145    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76fb72521cf26880273e72734546737e55eb329cad7d85261d27bddb255fe9c1"} err="failed to get container status \"76fb72521cf26880273e72734546737e55eb329cad7d85261d27bddb255fe9c1\": rpc error: code = NotFound desc = could not find container \"76fb72521cf26880273e72734546737e55eb329cad7d85261d27bddb255fe9c1\": container with ID starting with 76fb72521cf26880273e72734546737e55eb329cad7d85261d27bddb255fe9c1 not found: ID does not exist"
Jul 31 15:19:54.635405 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.635165    2559 scope.go:117] "RemoveContainer" containerID="fb16c5b5e38c108b9093cde4e47db987a8bdd3fe065f7efe4ed9e9d55fac0563"
Jul 31 15:19:54.635595 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:19:54.635572    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb16c5b5e38c108b9093cde4e47db987a8bdd3fe065f7efe4ed9e9d55fac0563\": container with ID starting with fb16c5b5e38c108b9093cde4e47db987a8bdd3fe065f7efe4ed9e9d55fac0563 not found: ID does not exist" containerID="fb16c5b5e38c108b9093cde4e47db987a8bdd3fe065f7efe4ed9e9d55fac0563"
Jul 31 15:19:54.635685 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.635604    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb16c5b5e38c108b9093cde4e47db987a8bdd3fe065f7efe4ed9e9d55fac0563"} err="failed to get container status \"fb16c5b5e38c108b9093cde4e47db987a8bdd3fe065f7efe4ed9e9d55fac0563\": rpc error: code = NotFound desc = could not find container \"fb16c5b5e38c108b9093cde4e47db987a8bdd3fe065f7efe4ed9e9d55fac0563\": container with ID starting with fb16c5b5e38c108b9093cde4e47db987a8bdd3fe065f7efe4ed9e9d55fac0563 not found: ID does not exist"
Jul 31 15:19:54.635685 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.635626    2559 scope.go:117] "RemoveContainer" containerID="58f4fab1a41e475e91a00bdaf189449c825748b52d4a8cfb73cacece36f2bebd"
Jul 31 15:19:54.636035 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:19:54.636007    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58f4fab1a41e475e91a00bdaf189449c825748b52d4a8cfb73cacece36f2bebd\": container with ID starting with 58f4fab1a41e475e91a00bdaf189449c825748b52d4a8cfb73cacece36f2bebd not found: ID does not exist" containerID="58f4fab1a41e475e91a00bdaf189449c825748b52d4a8cfb73cacece36f2bebd"
Jul 31 15:19:54.636091 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.636036    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58f4fab1a41e475e91a00bdaf189449c825748b52d4a8cfb73cacece36f2bebd"} err="failed to get container status \"58f4fab1a41e475e91a00bdaf189449c825748b52d4a8cfb73cacece36f2bebd\": rpc error: code = NotFound desc = could not find container \"58f4fab1a41e475e91a00bdaf189449c825748b52d4a8cfb73cacece36f2bebd\": container with ID starting with 58f4fab1a41e475e91a00bdaf189449c825748b52d4a8cfb73cacece36f2bebd not found: ID does not exist"
Jul 31 15:19:54.636952 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:54.636933    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z24l2"]
Jul 31 15:19:56.221490 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:56.221454    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52680717-9088-49bb-95c9-d55a0a38bd59" path="/var/lib/kubelet/pods/52680717-9088-49bb-95c9-d55a0a38bd59/volumes"
Jul 31 15:19:59.616250 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:59.616221    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"
Jul 31 15:19:59.616695 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:19:59.616654    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr" podUID="cd08fe2e-e0e6-473b-a431-e03e3441d28e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 31 15:20:03.458694 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:03.458657    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9" podUID="073437ef-93ac-4ef0-85e4-8e8c495c8d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 31 15:20:03.459163 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:03.458668    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628" podUID="872b0fc9-0302-4428-8784-a0a1e443db3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 31 15:20:09.617203 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:09.617159    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr" podUID="cd08fe2e-e0e6-473b-a431-e03e3441d28e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 31 15:20:13.458756 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:13.458697    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628" podUID="872b0fc9-0302-4428-8784-a0a1e443db3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 31 15:20:13.459146 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:13.458697    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9" podUID="073437ef-93ac-4ef0-85e4-8e8c495c8d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 31 15:20:19.617452 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:19.617417    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr" podUID="cd08fe2e-e0e6-473b-a431-e03e3441d28e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 31 15:20:23.458624 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:23.458583    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9" podUID="073437ef-93ac-4ef0-85e4-8e8c495c8d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 31 15:20:23.459050 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:23.458583    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628" podUID="872b0fc9-0302-4428-8784-a0a1e443db3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 31 15:20:28.226278 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:28.226233    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628" podUID="872b0fc9-0302-4428-8784-a0a1e443db3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 31 15:20:29.216901 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:29.216860    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9" podUID="073437ef-93ac-4ef0-85e4-8e8c495c8d7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 31 15:20:29.616962 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:29.616923    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr" podUID="cd08fe2e-e0e6-473b-a431-e03e3441d28e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 31 15:20:38.224164 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:38.224132    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"
Jul 31 15:20:39.217928 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:39.217894    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"
Jul 31 15:20:39.617680 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:39.617638    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr" podUID="cd08fe2e-e0e6-473b-a431-e03e3441d28e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 31 15:20:41.234995 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:41.234964    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"]
Jul 31 15:20:41.235381 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:41.235171    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9" podUID="073437ef-93ac-4ef0-85e4-8e8c495c8d7d" containerName="kserve-container" containerID="cri-o://a01846b294367dbf2461181aff031dbea65de5b6a552a9cfe4ba0c3cfb64af31" gracePeriod=30
Jul 31 15:20:41.235381 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:41.235317    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9" podUID="073437ef-93ac-4ef0-85e4-8e8c495c8d7d" containerName="kube-rbac-proxy" containerID="cri-o://cef77ccb09caa9e7f3119582cb58e44606478e309a24f8fd9ebd366fd3e8f75c" gracePeriod=30
Jul 31 15:20:41.803867 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:41.803837    2559 generic.go:356] "Generic (PLEG): container finished" podID="073437ef-93ac-4ef0-85e4-8e8c495c8d7d" containerID="cef77ccb09caa9e7f3119582cb58e44606478e309a24f8fd9ebd366fd3e8f75c" exitCode=2
Jul 31 15:20:41.804083 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:41.803938    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9" event={"ID":"073437ef-93ac-4ef0-85e4-8e8c495c8d7d","Type":"ContainerDied","Data":"cef77ccb09caa9e7f3119582cb58e44606478e309a24f8fd9ebd366fd3e8f75c"}
Jul 31 15:20:43.453539 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:43.453493    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9" podUID="073437ef-93ac-4ef0-85e4-8e8c495c8d7d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jul 31 15:20:46.073465 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.073439    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"
Jul 31 15:20:46.146790 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.146707    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bk6t2\" (UniqueName: \"kubernetes.io/projected/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-kube-api-access-bk6t2\") pod \"073437ef-93ac-4ef0-85e4-8e8c495c8d7d\" (UID: \"073437ef-93ac-4ef0-85e4-8e8c495c8d7d\") "
Jul 31 15:20:46.146790 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.146760    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-cabundle-cert\") pod \"073437ef-93ac-4ef0-85e4-8e8c495c8d7d\" (UID: \"073437ef-93ac-4ef0-85e4-8e8c495c8d7d\") "
Jul 31 15:20:46.146790 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.146784    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-kserve-provision-location\") pod \"073437ef-93ac-4ef0-85e4-8e8c495c8d7d\" (UID: \"073437ef-93ac-4ef0-85e4-8e8c495c8d7d\") "
Jul 31 15:20:46.147054 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.146816    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-kube-rbac-proxy-sar-config\") pod \"073437ef-93ac-4ef0-85e4-8e8c495c8d7d\" (UID: \"073437ef-93ac-4ef0-85e4-8e8c495c8d7d\") "
Jul 31 15:20:46.147054 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.146859    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-proxy-tls\") pod \"073437ef-93ac-4ef0-85e4-8e8c495c8d7d\" (UID: \"073437ef-93ac-4ef0-85e4-8e8c495c8d7d\") "
Jul 31 15:20:46.147162 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.147137    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "073437ef-93ac-4ef0-85e4-8e8c495c8d7d" (UID: "073437ef-93ac-4ef0-85e4-8e8c495c8d7d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:20:46.147219 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.147169    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "073437ef-93ac-4ef0-85e4-8e8c495c8d7d" (UID: "073437ef-93ac-4ef0-85e4-8e8c495c8d7d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:20:46.147219 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.147178    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "073437ef-93ac-4ef0-85e4-8e8c495c8d7d" (UID: "073437ef-93ac-4ef0-85e4-8e8c495c8d7d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:20:46.148894 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.148869    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-kube-api-access-bk6t2" (OuterVolumeSpecName: "kube-api-access-bk6t2") pod "073437ef-93ac-4ef0-85e4-8e8c495c8d7d" (UID: "073437ef-93ac-4ef0-85e4-8e8c495c8d7d"). InnerVolumeSpecName "kube-api-access-bk6t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:20:46.149008 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.148987    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "073437ef-93ac-4ef0-85e4-8e8c495c8d7d" (UID: "073437ef-93ac-4ef0-85e4-8e8c495c8d7d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:20:46.248218 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.248190    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:20:46.248218 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.248215    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:20:46.248375 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.248226    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk6t2\" (UniqueName: \"kubernetes.io/projected/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-kube-api-access-bk6t2\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:20:46.248375 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.248235    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:20:46.248375 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.248243    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/073437ef-93ac-4ef0-85e4-8e8c495c8d7d-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:20:46.827022 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.826989    2559 generic.go:356] "Generic (PLEG): container finished" podID="073437ef-93ac-4ef0-85e4-8e8c495c8d7d" containerID="a01846b294367dbf2461181aff031dbea65de5b6a552a9cfe4ba0c3cfb64af31" exitCode=0
Jul 31 15:20:46.827192 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.827047    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"
Jul 31 15:20:46.827192 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.827074    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9" event={"ID":"073437ef-93ac-4ef0-85e4-8e8c495c8d7d","Type":"ContainerDied","Data":"a01846b294367dbf2461181aff031dbea65de5b6a552a9cfe4ba0c3cfb64af31"}
Jul 31 15:20:46.827192 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.827136    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9" event={"ID":"073437ef-93ac-4ef0-85e4-8e8c495c8d7d","Type":"ContainerDied","Data":"c3194382b4040d2002240c4fd21ef0858cb3b359cbc53642375f2f0e871b0b9d"}
Jul 31 15:20:46.827192 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.827162    2559 scope.go:117] "RemoveContainer" containerID="cef77ccb09caa9e7f3119582cb58e44606478e309a24f8fd9ebd366fd3e8f75c"
Jul 31 15:20:46.835245 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.835228    2559 scope.go:117] "RemoveContainer" containerID="a01846b294367dbf2461181aff031dbea65de5b6a552a9cfe4ba0c3cfb64af31"
Jul 31 15:20:46.842642 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.842623    2559 scope.go:117] "RemoveContainer" containerID="9bf67e069a73d829e02fcb004623c3c9418660959cbff6e16fe09589f79cbaf4"
Jul 31 15:20:46.845480 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.845453    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"]
Jul 31 15:20:46.849917 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.849895    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cd9bb4b6b-5dsc9"]
Jul 31 15:20:46.850725 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.850710    2559 scope.go:117] "RemoveContainer" containerID="cef77ccb09caa9e7f3119582cb58e44606478e309a24f8fd9ebd366fd3e8f75c"
Jul 31 15:20:46.851010 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:20:46.850982    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cef77ccb09caa9e7f3119582cb58e44606478e309a24f8fd9ebd366fd3e8f75c\": container with ID starting with cef77ccb09caa9e7f3119582cb58e44606478e309a24f8fd9ebd366fd3e8f75c not found: ID does not exist" containerID="cef77ccb09caa9e7f3119582cb58e44606478e309a24f8fd9ebd366fd3e8f75c"
Jul 31 15:20:46.851076 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.851022    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cef77ccb09caa9e7f3119582cb58e44606478e309a24f8fd9ebd366fd3e8f75c"} err="failed to get container status \"cef77ccb09caa9e7f3119582cb58e44606478e309a24f8fd9ebd366fd3e8f75c\": rpc error: code = NotFound desc = could not find container \"cef77ccb09caa9e7f3119582cb58e44606478e309a24f8fd9ebd366fd3e8f75c\": container with ID starting with cef77ccb09caa9e7f3119582cb58e44606478e309a24f8fd9ebd366fd3e8f75c not found: ID does not exist"
Jul 31 15:20:46.851076 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.851045    2559 scope.go:117] "RemoveContainer" containerID="a01846b294367dbf2461181aff031dbea65de5b6a552a9cfe4ba0c3cfb64af31"
Jul 31 15:20:46.851278 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:20:46.851261    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a01846b294367dbf2461181aff031dbea65de5b6a552a9cfe4ba0c3cfb64af31\": container with ID starting with a01846b294367dbf2461181aff031dbea65de5b6a552a9cfe4ba0c3cfb64af31 not found: ID does not exist" containerID="a01846b294367dbf2461181aff031dbea65de5b6a552a9cfe4ba0c3cfb64af31"
Jul 31 15:20:46.851323 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.851284    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a01846b294367dbf2461181aff031dbea65de5b6a552a9cfe4ba0c3cfb64af31"} err="failed to get container status \"a01846b294367dbf2461181aff031dbea65de5b6a552a9cfe4ba0c3cfb64af31\": rpc error: code = NotFound desc = could not find container \"a01846b294367dbf2461181aff031dbea65de5b6a552a9cfe4ba0c3cfb64af31\": container with ID starting with a01846b294367dbf2461181aff031dbea65de5b6a552a9cfe4ba0c3cfb64af31 not found: ID does not exist"
Jul 31 15:20:46.851323 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.851300    2559 scope.go:117] "RemoveContainer" containerID="9bf67e069a73d829e02fcb004623c3c9418660959cbff6e16fe09589f79cbaf4"
Jul 31 15:20:46.851477 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:20:46.851463    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bf67e069a73d829e02fcb004623c3c9418660959cbff6e16fe09589f79cbaf4\": container with ID starting with 9bf67e069a73d829e02fcb004623c3c9418660959cbff6e16fe09589f79cbaf4 not found: ID does not exist" containerID="9bf67e069a73d829e02fcb004623c3c9418660959cbff6e16fe09589f79cbaf4"
Jul 31 15:20:46.851518 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:46.851480    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf67e069a73d829e02fcb004623c3c9418660959cbff6e16fe09589f79cbaf4"} err="failed to get container status \"9bf67e069a73d829e02fcb004623c3c9418660959cbff6e16fe09589f79cbaf4\": rpc error: code = NotFound desc = could not find container \"9bf67e069a73d829e02fcb004623c3c9418660959cbff6e16fe09589f79cbaf4\": container with ID starting with 9bf67e069a73d829e02fcb004623c3c9418660959cbff6e16fe09589f79cbaf4 not found: ID does not exist"
Jul 31 15:20:48.221180 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:48.221127    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="073437ef-93ac-4ef0-85e4-8e8c495c8d7d" path="/var/lib/kubelet/pods/073437ef-93ac-4ef0-85e4-8e8c495c8d7d/volumes"
Jul 31 15:20:49.616965 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:49.616928    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr" podUID="cd08fe2e-e0e6-473b-a431-e03e3441d28e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 31 15:20:59.617597 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:20:59.617554    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr" podUID="cd08fe2e-e0e6-473b-a431-e03e3441d28e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 31 15:21:09.617450 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:09.617421    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"
Jul 31 15:21:17.459671 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:17.459590    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"]
Jul 31 15:21:17.460099 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:17.459856    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr" podUID="cd08fe2e-e0e6-473b-a431-e03e3441d28e" containerName="kserve-container" containerID="cri-o://08d34f9f31844105f10ac9fe05ca6477b46044d933a9c99faf24d51def59a174" gracePeriod=30
Jul 31 15:21:17.460099 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:17.459882    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr" podUID="cd08fe2e-e0e6-473b-a431-e03e3441d28e" containerName="kube-rbac-proxy" containerID="cri-o://56db66c16f2353b0faec99bdeca03998682a65d707ac64e0c0d43aae17c68fa6" gracePeriod=30
Jul 31 15:21:17.943414 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:17.943380    2559 generic.go:356] "Generic (PLEG): container finished" podID="cd08fe2e-e0e6-473b-a431-e03e3441d28e" containerID="56db66c16f2353b0faec99bdeca03998682a65d707ac64e0c0d43aae17c68fa6" exitCode=2
Jul 31 15:21:17.943602 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:17.943457    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr" event={"ID":"cd08fe2e-e0e6-473b-a431-e03e3441d28e","Type":"ContainerDied","Data":"56db66c16f2353b0faec99bdeca03998682a65d707ac64e0c0d43aae17c68fa6"}
Jul 31 15:21:19.612683 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:19.612635    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr" podUID="cd08fe2e-e0e6-473b-a431-e03e3441d28e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Jul 31 15:21:19.616926 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:19.616901    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr" podUID="cd08fe2e-e0e6-473b-a431-e03e3441d28e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 31 15:21:22.293250 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.293225    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"
Jul 31 15:21:22.434006 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.433968    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd08fe2e-e0e6-473b-a431-e03e3441d28e-kserve-provision-location\") pod \"cd08fe2e-e0e6-473b-a431-e03e3441d28e\" (UID: \"cd08fe2e-e0e6-473b-a431-e03e3441d28e\") "
Jul 31 15:21:22.434225 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.434028    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd08fe2e-e0e6-473b-a431-e03e3441d28e-proxy-tls\") pod \"cd08fe2e-e0e6-473b-a431-e03e3441d28e\" (UID: \"cd08fe2e-e0e6-473b-a431-e03e3441d28e\") "
Jul 31 15:21:22.434225 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.434060    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd08fe2e-e0e6-473b-a431-e03e3441d28e-kube-rbac-proxy-sar-config\") pod \"cd08fe2e-e0e6-473b-a431-e03e3441d28e\" (UID: \"cd08fe2e-e0e6-473b-a431-e03e3441d28e\") "
Jul 31 15:21:22.434225 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.434113    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd08fe2e-e0e6-473b-a431-e03e3441d28e-cabundle-cert\") pod \"cd08fe2e-e0e6-473b-a431-e03e3441d28e\" (UID: \"cd08fe2e-e0e6-473b-a431-e03e3441d28e\") "
Jul 31 15:21:22.434419 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.434247    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npfxw\" (UniqueName: \"kubernetes.io/projected/cd08fe2e-e0e6-473b-a431-e03e3441d28e-kube-api-access-npfxw\") pod \"cd08fe2e-e0e6-473b-a431-e03e3441d28e\" (UID: \"cd08fe2e-e0e6-473b-a431-e03e3441d28e\") "
Jul 31 15:21:22.434479 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.434424    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd08fe2e-e0e6-473b-a431-e03e3441d28e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cd08fe2e-e0e6-473b-a431-e03e3441d28e" (UID: "cd08fe2e-e0e6-473b-a431-e03e3441d28e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:21:22.434479 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.434454    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd08fe2e-e0e6-473b-a431-e03e3441d28e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cd08fe2e-e0e6-473b-a431-e03e3441d28e" (UID: "cd08fe2e-e0e6-473b-a431-e03e3441d28e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:21:22.434562 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.434478    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd08fe2e-e0e6-473b-a431-e03e3441d28e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cd08fe2e-e0e6-473b-a431-e03e3441d28e" (UID: "cd08fe2e-e0e6-473b-a431-e03e3441d28e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:21:22.434562 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.434528    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd08fe2e-e0e6-473b-a431-e03e3441d28e-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:21:22.434562 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.434547    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd08fe2e-e0e6-473b-a431-e03e3441d28e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:21:22.436374 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.436347    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd08fe2e-e0e6-473b-a431-e03e3441d28e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cd08fe2e-e0e6-473b-a431-e03e3441d28e" (UID: "cd08fe2e-e0e6-473b-a431-e03e3441d28e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:21:22.436374 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.436368    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd08fe2e-e0e6-473b-a431-e03e3441d28e-kube-api-access-npfxw" (OuterVolumeSpecName: "kube-api-access-npfxw") pod "cd08fe2e-e0e6-473b-a431-e03e3441d28e" (UID: "cd08fe2e-e0e6-473b-a431-e03e3441d28e"). InnerVolumeSpecName "kube-api-access-npfxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:21:22.535645 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.535607    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd08fe2e-e0e6-473b-a431-e03e3441d28e-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:21:22.535645 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.535638    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npfxw\" (UniqueName: \"kubernetes.io/projected/cd08fe2e-e0e6-473b-a431-e03e3441d28e-kube-api-access-npfxw\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:21:22.535645 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.535649    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd08fe2e-e0e6-473b-a431-e03e3441d28e-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:21:22.637980 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.637941    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"]
Jul 31 15:21:22.638228 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.638216    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52680717-9088-49bb-95c9-d55a0a38bd59" containerName="kube-rbac-proxy"
Jul 31 15:21:22.638274 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.638229    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="52680717-9088-49bb-95c9-d55a0a38bd59" containerName="kube-rbac-proxy"
Jul 31 15:21:22.638274 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.638240    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="073437ef-93ac-4ef0-85e4-8e8c495c8d7d" containerName="kserve-container"
Jul 31 15:21:22.638274 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.638246    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="073437ef-93ac-4ef0-85e4-8e8c495c8d7d" containerName="kserve-container"
Jul 31 15:21:22.638274 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.638254    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52680717-9088-49bb-95c9-d55a0a38bd59" containerName="storage-initializer"
Jul 31 15:21:22.638274 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.638259    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="52680717-9088-49bb-95c9-d55a0a38bd59" containerName="storage-initializer"
Jul 31 15:21:22.638274 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.638268    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="073437ef-93ac-4ef0-85e4-8e8c495c8d7d" containerName="storage-initializer"
Jul 31 15:21:22.638274 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.638273    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="073437ef-93ac-4ef0-85e4-8e8c495c8d7d" containerName="storage-initializer"
Jul 31 15:21:22.638488 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.638281    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd08fe2e-e0e6-473b-a431-e03e3441d28e" containerName="storage-initializer"
Jul 31 15:21:22.638488 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.638286    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd08fe2e-e0e6-473b-a431-e03e3441d28e" containerName="storage-initializer"
Jul 31 15:21:22.638488 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.638292    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="073437ef-93ac-4ef0-85e4-8e8c495c8d7d" containerName="kube-rbac-proxy"
Jul 31 15:21:22.638488 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.638297    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="073437ef-93ac-4ef0-85e4-8e8c495c8d7d" containerName="kube-rbac-proxy"
Jul 31 15:21:22.638488 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.638303    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd08fe2e-e0e6-473b-a431-e03e3441d28e" containerName="kube-rbac-proxy"
Jul 31 15:21:22.638488 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.638308    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd08fe2e-e0e6-473b-a431-e03e3441d28e" containerName="kube-rbac-proxy"
Jul 31 15:21:22.638488 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.638314    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd08fe2e-e0e6-473b-a431-e03e3441d28e" containerName="kserve-container"
Jul 31 15:21:22.638488 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.638319    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd08fe2e-e0e6-473b-a431-e03e3441d28e" containerName="kserve-container"
Jul 31 15:21:22.638488 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.638326    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52680717-9088-49bb-95c9-d55a0a38bd59" containerName="kserve-container"
Jul 31 15:21:22.638488 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.638331    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="52680717-9088-49bb-95c9-d55a0a38bd59" containerName="kserve-container"
Jul 31 15:21:22.638488 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.638389    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd08fe2e-e0e6-473b-a431-e03e3441d28e" containerName="kube-rbac-proxy"
Jul 31 15:21:22.638488 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.638402    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="52680717-9088-49bb-95c9-d55a0a38bd59" containerName="kube-rbac-proxy"
Jul 31 15:21:22.638488 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.638408    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="073437ef-93ac-4ef0-85e4-8e8c495c8d7d" containerName="kube-rbac-proxy"
Jul 31 15:21:22.638488 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.638415    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="073437ef-93ac-4ef0-85e4-8e8c495c8d7d" containerName="kserve-container"
Jul 31 15:21:22.638488 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.638420    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd08fe2e-e0e6-473b-a431-e03e3441d28e" containerName="kserve-container"
Jul 31 15:21:22.638488 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.638425    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="52680717-9088-49bb-95c9-d55a0a38bd59" containerName="kserve-container"
Jul 31 15:21:22.641673 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.641657    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"
Jul 31 15:21:22.644222 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.644195    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Jul 31 15:21:22.644371 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.644228    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Jul 31 15:21:22.651776 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.651753    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"]
Jul 31 15:21:22.738001 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.737906    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rgqw\" (UniqueName: \"kubernetes.io/projected/77d625f7-0327-478a-b7e4-4334a02942d2-kube-api-access-9rgqw\") pod \"isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq\" (UID: \"77d625f7-0327-478a-b7e4-4334a02942d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"
Jul 31 15:21:22.738001 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.737960    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77d625f7-0327-478a-b7e4-4334a02942d2-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq\" (UID: \"77d625f7-0327-478a-b7e4-4334a02942d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"
Jul 31 15:21:22.738001 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.737979    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77d625f7-0327-478a-b7e4-4334a02942d2-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq\" (UID: \"77d625f7-0327-478a-b7e4-4334a02942d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"
Jul 31 15:21:22.738239 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.738016    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77d625f7-0327-478a-b7e4-4334a02942d2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq\" (UID: \"77d625f7-0327-478a-b7e4-4334a02942d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"
Jul 31 15:21:22.738239 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.738035    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77d625f7-0327-478a-b7e4-4334a02942d2-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq\" (UID: \"77d625f7-0327-478a-b7e4-4334a02942d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"
Jul 31 15:21:22.838459 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.838423    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rgqw\" (UniqueName: \"kubernetes.io/projected/77d625f7-0327-478a-b7e4-4334a02942d2-kube-api-access-9rgqw\") pod \"isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq\" (UID: \"77d625f7-0327-478a-b7e4-4334a02942d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"
Jul 31 15:21:22.838459 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.838463    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77d625f7-0327-478a-b7e4-4334a02942d2-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq\" (UID: \"77d625f7-0327-478a-b7e4-4334a02942d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"
Jul 31 15:21:22.838703 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.838489    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77d625f7-0327-478a-b7e4-4334a02942d2-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq\" (UID: \"77d625f7-0327-478a-b7e4-4334a02942d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"
Jul 31 15:21:22.838703 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.838508    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77d625f7-0327-478a-b7e4-4334a02942d2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq\" (UID: \"77d625f7-0327-478a-b7e4-4334a02942d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"
Jul 31 15:21:22.838703 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.838528    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77d625f7-0327-478a-b7e4-4334a02942d2-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq\" (UID: \"77d625f7-0327-478a-b7e4-4334a02942d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"
Jul 31 15:21:22.838957 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.838936    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77d625f7-0327-478a-b7e4-4334a02942d2-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq\" (UID: \"77d625f7-0327-478a-b7e4-4334a02942d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"
Jul 31 15:21:22.839246 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.839227    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77d625f7-0327-478a-b7e4-4334a02942d2-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq\" (UID: \"77d625f7-0327-478a-b7e4-4334a02942d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"
Jul 31 15:21:22.839399 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.839279    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77d625f7-0327-478a-b7e4-4334a02942d2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq\" (UID: \"77d625f7-0327-478a-b7e4-4334a02942d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"
Jul 31 15:21:22.841450 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.841421    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77d625f7-0327-478a-b7e4-4334a02942d2-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq\" (UID: \"77d625f7-0327-478a-b7e4-4334a02942d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"
Jul 31 15:21:22.846991 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.846970    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rgqw\" (UniqueName: \"kubernetes.io/projected/77d625f7-0327-478a-b7e4-4334a02942d2-kube-api-access-9rgqw\") pod \"isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq\" (UID: \"77d625f7-0327-478a-b7e4-4334a02942d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"
Jul 31 15:21:22.952825 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.952785    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"
Jul 31 15:21:22.964799 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.964775    2559 generic.go:356] "Generic (PLEG): container finished" podID="cd08fe2e-e0e6-473b-a431-e03e3441d28e" containerID="08d34f9f31844105f10ac9fe05ca6477b46044d933a9c99faf24d51def59a174" exitCode=0
Jul 31 15:21:22.964900 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.964828    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"
Jul 31 15:21:22.964900 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.964854    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr" event={"ID":"cd08fe2e-e0e6-473b-a431-e03e3441d28e","Type":"ContainerDied","Data":"08d34f9f31844105f10ac9fe05ca6477b46044d933a9c99faf24d51def59a174"}
Jul 31 15:21:22.964989 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.964915    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr" event={"ID":"cd08fe2e-e0e6-473b-a431-e03e3441d28e","Type":"ContainerDied","Data":"16668d0ae9eb4d685f5633b82adf4e613644aee2291e86966015d0e4bc9983c7"}
Jul 31 15:21:22.964989 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.964943    2559 scope.go:117] "RemoveContainer" containerID="56db66c16f2353b0faec99bdeca03998682a65d707ac64e0c0d43aae17c68fa6"
Jul 31 15:21:22.973096 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.973074    2559 scope.go:117] "RemoveContainer" containerID="08d34f9f31844105f10ac9fe05ca6477b46044d933a9c99faf24d51def59a174"
Jul 31 15:21:22.982261 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.982242    2559 scope.go:117] "RemoveContainer" containerID="908c0b0dbd7015b9340a11f0d79cc8eb443058af33d7ba8fcaa22417a196ca1c"
Jul 31 15:21:22.990771 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.990680    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"]
Jul 31 15:21:22.992434 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.992416    2559 scope.go:117] "RemoveContainer" containerID="56db66c16f2353b0faec99bdeca03998682a65d707ac64e0c0d43aae17c68fa6"
Jul 31 15:21:22.992789 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:21:22.992758    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56db66c16f2353b0faec99bdeca03998682a65d707ac64e0c0d43aae17c68fa6\": container with ID starting with 56db66c16f2353b0faec99bdeca03998682a65d707ac64e0c0d43aae17c68fa6 not found: ID does not exist" containerID="56db66c16f2353b0faec99bdeca03998682a65d707ac64e0c0d43aae17c68fa6"
Jul 31 15:21:22.992909 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.992796    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56db66c16f2353b0faec99bdeca03998682a65d707ac64e0c0d43aae17c68fa6"} err="failed to get container status \"56db66c16f2353b0faec99bdeca03998682a65d707ac64e0c0d43aae17c68fa6\": rpc error: code = NotFound desc = could not find container \"56db66c16f2353b0faec99bdeca03998682a65d707ac64e0c0d43aae17c68fa6\": container with ID starting with 56db66c16f2353b0faec99bdeca03998682a65d707ac64e0c0d43aae17c68fa6 not found: ID does not exist"
Jul 31 15:21:22.992909 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.992818    2559 scope.go:117] "RemoveContainer" containerID="08d34f9f31844105f10ac9fe05ca6477b46044d933a9c99faf24d51def59a174"
Jul 31 15:21:22.993206 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:21:22.993181    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08d34f9f31844105f10ac9fe05ca6477b46044d933a9c99faf24d51def59a174\": container with ID starting with 08d34f9f31844105f10ac9fe05ca6477b46044d933a9c99faf24d51def59a174 not found: ID does not exist" containerID="08d34f9f31844105f10ac9fe05ca6477b46044d933a9c99faf24d51def59a174"
Jul 31 15:21:22.993281 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.993219    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08d34f9f31844105f10ac9fe05ca6477b46044d933a9c99faf24d51def59a174"} err="failed to get container status \"08d34f9f31844105f10ac9fe05ca6477b46044d933a9c99faf24d51def59a174\": rpc error: code = NotFound desc = could not find container \"08d34f9f31844105f10ac9fe05ca6477b46044d933a9c99faf24d51def59a174\": container with ID starting with 08d34f9f31844105f10ac9fe05ca6477b46044d933a9c99faf24d51def59a174 not found: ID does not exist"
Jul 31 15:21:22.993281 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.993246    2559 scope.go:117] "RemoveContainer" containerID="908c0b0dbd7015b9340a11f0d79cc8eb443058af33d7ba8fcaa22417a196ca1c"
Jul 31 15:21:22.993585 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:21:22.993568    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"908c0b0dbd7015b9340a11f0d79cc8eb443058af33d7ba8fcaa22417a196ca1c\": container with ID starting with 908c0b0dbd7015b9340a11f0d79cc8eb443058af33d7ba8fcaa22417a196ca1c not found: ID does not exist" containerID="908c0b0dbd7015b9340a11f0d79cc8eb443058af33d7ba8fcaa22417a196ca1c"
Jul 31 15:21:22.993651 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.993593    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"908c0b0dbd7015b9340a11f0d79cc8eb443058af33d7ba8fcaa22417a196ca1c"} err="failed to get container status \"908c0b0dbd7015b9340a11f0d79cc8eb443058af33d7ba8fcaa22417a196ca1c\": rpc error: code = NotFound desc = could not find container \"908c0b0dbd7015b9340a11f0d79cc8eb443058af33d7ba8fcaa22417a196ca1c\": container with ID starting with 908c0b0dbd7015b9340a11f0d79cc8eb443058af33d7ba8fcaa22417a196ca1c not found: ID does not exist"
Jul 31 15:21:22.996244 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:22.996224    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-6fd4dbf555-gr2jr"]
Jul 31 15:21:23.100452 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:23.100415    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"]
Jul 31 15:21:23.103075 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:21:23.103048    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d625f7_0327_478a_b7e4_4334a02942d2.slice/crio-3308bdbcf95f0e1004877e4beca4095c261db86bd780d9c5c249b584751fa425 WatchSource:0}: Error finding container 3308bdbcf95f0e1004877e4beca4095c261db86bd780d9c5c249b584751fa425: Status 404 returned error can't find the container with id 3308bdbcf95f0e1004877e4beca4095c261db86bd780d9c5c249b584751fa425
Jul 31 15:21:23.970932 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:23.970892    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq" event={"ID":"77d625f7-0327-478a-b7e4-4334a02942d2","Type":"ContainerStarted","Data":"e3c95b305fa645320c11526b74a959128d31c54ed5bb35e780d5aaf298b271c8"}
Jul 31 15:21:23.971388 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:23.970953    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq" event={"ID":"77d625f7-0327-478a-b7e4-4334a02942d2","Type":"ContainerStarted","Data":"3308bdbcf95f0e1004877e4beca4095c261db86bd780d9c5c249b584751fa425"}
Jul 31 15:21:24.221231 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:24.221158    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd08fe2e-e0e6-473b-a431-e03e3441d28e" path="/var/lib/kubelet/pods/cd08fe2e-e0e6-473b-a431-e03e3441d28e/volumes"
Jul 31 15:21:26.984206 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:26.984179    2559 generic.go:356] "Generic (PLEG): container finished" podID="77d625f7-0327-478a-b7e4-4334a02942d2" containerID="e3c95b305fa645320c11526b74a959128d31c54ed5bb35e780d5aaf298b271c8" exitCode=0
Jul 31 15:21:26.984551 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:26.984235    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq" event={"ID":"77d625f7-0327-478a-b7e4-4334a02942d2","Type":"ContainerDied","Data":"e3c95b305fa645320c11526b74a959128d31c54ed5bb35e780d5aaf298b271c8"}
Jul 31 15:21:27.989815 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:27.989780    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq" event={"ID":"77d625f7-0327-478a-b7e4-4334a02942d2","Type":"ContainerStarted","Data":"cdf1f5f1391f4ff47798a28f0db54f062b2afce6c79698afd3e253e2fa144ad9"}
Jul 31 15:21:27.990219 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:27.989836    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq" event={"ID":"77d625f7-0327-478a-b7e4-4334a02942d2","Type":"ContainerStarted","Data":"c68da27eab27ac07f49aadbf05f22be3976d3c0c423bca5e33ad26e2629510bd"}
Jul 31 15:21:27.990219 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:27.990025    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"
Jul 31 15:21:28.011974 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:28.011925    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq" podStartSLOduration=2.219218962 podStartE2EDuration="6.011908447s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.792689485s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:21:28.010120661 +0000 UTC m=+1984.321678295" watchObservedRunningTime="2026-07-31 15:21:28.011908447 +0000 UTC m=+1984.323466069"
Jul 31 15:21:28.993521 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:28.993488    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"
Jul 31 15:21:28.994810 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:28.994782    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq" podUID="77d625f7-0327-478a-b7e4-4334a02942d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 31 15:21:29.997244 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:29.997203    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq" podUID="77d625f7-0327-478a-b7e4-4334a02942d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 31 15:21:35.003561 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:35.003531    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"
Jul 31 15:21:35.004162 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:35.004136    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq" podUID="77d625f7-0327-478a-b7e4-4334a02942d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 31 15:21:45.004855 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:45.004818    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq" podUID="77d625f7-0327-478a-b7e4-4334a02942d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 31 15:21:55.004888 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:21:55.004843    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq" podUID="77d625f7-0327-478a-b7e4-4334a02942d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 31 15:22:05.004411 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:05.004367    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq" podUID="77d625f7-0327-478a-b7e4-4334a02942d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 31 15:22:15.004401 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:15.004360    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq" podUID="77d625f7-0327-478a-b7e4-4334a02942d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 31 15:22:21.467469 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:21.467436    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"]
Jul 31 15:22:21.467930 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:21.467703    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628" podUID="872b0fc9-0302-4428-8784-a0a1e443db3f" containerName="kserve-container" containerID="cri-o://5803bc37185849965f862c3a30d267556f57c348df5afa55c887e85cb047a440" gracePeriod=30
Jul 31 15:22:21.467930 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:21.467766    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628" podUID="872b0fc9-0302-4428-8784-a0a1e443db3f" containerName="kube-rbac-proxy" containerID="cri-o://dddf5141e93cc050ef660baeb29d668f584572a5e91c19d4c4943f0bff32e0c9" gracePeriod=30
Jul 31 15:22:22.191132 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:22.191098    2559 generic.go:356] "Generic (PLEG): container finished" podID="872b0fc9-0302-4428-8784-a0a1e443db3f" containerID="dddf5141e93cc050ef660baeb29d668f584572a5e91c19d4c4943f0bff32e0c9" exitCode=2
Jul 31 15:22:22.191318 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:22.191162    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628" event={"ID":"872b0fc9-0302-4428-8784-a0a1e443db3f","Type":"ContainerDied","Data":"dddf5141e93cc050ef660baeb29d668f584572a5e91c19d4c4943f0bff32e0c9"}
Jul 31 15:22:23.454029 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:23.453992    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628" podUID="872b0fc9-0302-4428-8784-a0a1e443db3f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Jul 31 15:22:25.004202 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:25.004163    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq" podUID="77d625f7-0327-478a-b7e4-4334a02942d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 31 15:22:26.420460 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.420435    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"
Jul 31 15:22:26.502238 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.502207    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/872b0fc9-0302-4428-8784-a0a1e443db3f-proxy-tls\") pod \"872b0fc9-0302-4428-8784-a0a1e443db3f\" (UID: \"872b0fc9-0302-4428-8784-a0a1e443db3f\") "
Jul 31 15:22:26.502429 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.502254    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/872b0fc9-0302-4428-8784-a0a1e443db3f-kube-rbac-proxy-sar-config\") pod \"872b0fc9-0302-4428-8784-a0a1e443db3f\" (UID: \"872b0fc9-0302-4428-8784-a0a1e443db3f\") "
Jul 31 15:22:26.502429 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.502285    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfnxv\" (UniqueName: \"kubernetes.io/projected/872b0fc9-0302-4428-8784-a0a1e443db3f-kube-api-access-zfnxv\") pod \"872b0fc9-0302-4428-8784-a0a1e443db3f\" (UID: \"872b0fc9-0302-4428-8784-a0a1e443db3f\") "
Jul 31 15:22:26.502429 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.502410    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/872b0fc9-0302-4428-8784-a0a1e443db3f-cabundle-cert\") pod \"872b0fc9-0302-4428-8784-a0a1e443db3f\" (UID: \"872b0fc9-0302-4428-8784-a0a1e443db3f\") "
Jul 31 15:22:26.502618 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.502450    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/872b0fc9-0302-4428-8784-a0a1e443db3f-kserve-provision-location\") pod \"872b0fc9-0302-4428-8784-a0a1e443db3f\" (UID: \"872b0fc9-0302-4428-8784-a0a1e443db3f\") "
Jul 31 15:22:26.502779 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.502722    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/872b0fc9-0302-4428-8784-a0a1e443db3f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "872b0fc9-0302-4428-8784-a0a1e443db3f" (UID: "872b0fc9-0302-4428-8784-a0a1e443db3f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:22:26.502897 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.502817    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872b0fc9-0302-4428-8784-a0a1e443db3f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "872b0fc9-0302-4428-8784-a0a1e443db3f" (UID: "872b0fc9-0302-4428-8784-a0a1e443db3f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:22:26.502897 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.502838    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/872b0fc9-0302-4428-8784-a0a1e443db3f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "872b0fc9-0302-4428-8784-a0a1e443db3f" (UID: "872b0fc9-0302-4428-8784-a0a1e443db3f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:22:26.504934 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.504909    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/872b0fc9-0302-4428-8784-a0a1e443db3f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "872b0fc9-0302-4428-8784-a0a1e443db3f" (UID: "872b0fc9-0302-4428-8784-a0a1e443db3f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:22:26.504934 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.504922    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872b0fc9-0302-4428-8784-a0a1e443db3f-kube-api-access-zfnxv" (OuterVolumeSpecName: "kube-api-access-zfnxv") pod "872b0fc9-0302-4428-8784-a0a1e443db3f" (UID: "872b0fc9-0302-4428-8784-a0a1e443db3f"). InnerVolumeSpecName "kube-api-access-zfnxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:22:26.602970 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.602944    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/872b0fc9-0302-4428-8784-a0a1e443db3f-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:22:26.602970 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.602969    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/872b0fc9-0302-4428-8784-a0a1e443db3f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:22:26.603171 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.602980    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfnxv\" (UniqueName: \"kubernetes.io/projected/872b0fc9-0302-4428-8784-a0a1e443db3f-kube-api-access-zfnxv\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:22:26.603171 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.602989    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/872b0fc9-0302-4428-8784-a0a1e443db3f-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:22:26.603171 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.602997    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/872b0fc9-0302-4428-8784-a0a1e443db3f-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:22:26.775069 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.775038    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"]
Jul 31 15:22:26.775311 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.775299    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="872b0fc9-0302-4428-8784-a0a1e443db3f" containerName="storage-initializer"
Jul 31 15:22:26.775357 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.775313    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="872b0fc9-0302-4428-8784-a0a1e443db3f" containerName="storage-initializer"
Jul 31 15:22:26.775357 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.775329    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="872b0fc9-0302-4428-8784-a0a1e443db3f" containerName="kube-rbac-proxy"
Jul 31 15:22:26.775357 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.775335    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="872b0fc9-0302-4428-8784-a0a1e443db3f" containerName="kube-rbac-proxy"
Jul 31 15:22:26.775357 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.775348    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="872b0fc9-0302-4428-8784-a0a1e443db3f" containerName="kserve-container"
Jul 31 15:22:26.775357 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.775354    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="872b0fc9-0302-4428-8784-a0a1e443db3f" containerName="kserve-container"
Jul 31 15:22:26.775512 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.775398    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="872b0fc9-0302-4428-8784-a0a1e443db3f" containerName="kserve-container"
Jul 31 15:22:26.775512 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.775405    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="872b0fc9-0302-4428-8784-a0a1e443db3f" containerName="kube-rbac-proxy"
Jul 31 15:22:26.778544 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.778526    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"
Jul 31 15:22:26.782285 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.782252    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Jul 31 15:22:26.782414 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.782286    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 31 15:22:26.815311 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.813021    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"]
Jul 31 15:22:26.905664 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.905585    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/086dab96-a305-4bcd-8d62-b36f73d429c0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-758954f9f-w8hp6\" (UID: \"086dab96-a305-4bcd-8d62-b36f73d429c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"
Jul 31 15:22:26.905664 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.905623    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5bkn\" (UniqueName: \"kubernetes.io/projected/086dab96-a305-4bcd-8d62-b36f73d429c0-kube-api-access-s5bkn\") pod \"isvc-canary-rollback-v2-predictor-758954f9f-w8hp6\" (UID: \"086dab96-a305-4bcd-8d62-b36f73d429c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"
Jul 31 15:22:26.905664 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.905659    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/086dab96-a305-4bcd-8d62-b36f73d429c0-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-758954f9f-w8hp6\" (UID: \"086dab96-a305-4bcd-8d62-b36f73d429c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"
Jul 31 15:22:26.905914 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.905692    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/086dab96-a305-4bcd-8d62-b36f73d429c0-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-758954f9f-w8hp6\" (UID: \"086dab96-a305-4bcd-8d62-b36f73d429c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"
Jul 31 15:22:26.905914 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:26.905709    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/086dab96-a305-4bcd-8d62-b36f73d429c0-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-758954f9f-w8hp6\" (UID: \"086dab96-a305-4bcd-8d62-b36f73d429c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"
Jul 31 15:22:27.006636 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:27.006595    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5bkn\" (UniqueName: \"kubernetes.io/projected/086dab96-a305-4bcd-8d62-b36f73d429c0-kube-api-access-s5bkn\") pod \"isvc-canary-rollback-v2-predictor-758954f9f-w8hp6\" (UID: \"086dab96-a305-4bcd-8d62-b36f73d429c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"
Jul 31 15:22:27.006846 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:27.006643    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/086dab96-a305-4bcd-8d62-b36f73d429c0-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-758954f9f-w8hp6\" (UID: \"086dab96-a305-4bcd-8d62-b36f73d429c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"
Jul 31 15:22:27.006846 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:27.006685    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/086dab96-a305-4bcd-8d62-b36f73d429c0-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-758954f9f-w8hp6\" (UID: \"086dab96-a305-4bcd-8d62-b36f73d429c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"
Jul 31 15:22:27.006846 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:27.006713    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/086dab96-a305-4bcd-8d62-b36f73d429c0-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-758954f9f-w8hp6\" (UID: \"086dab96-a305-4bcd-8d62-b36f73d429c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"
Jul 31 15:22:27.006846 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:27.006799    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/086dab96-a305-4bcd-8d62-b36f73d429c0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-758954f9f-w8hp6\" (UID: \"086dab96-a305-4bcd-8d62-b36f73d429c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"
Jul 31 15:22:27.007125 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:27.007102    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/086dab96-a305-4bcd-8d62-b36f73d429c0-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-758954f9f-w8hp6\" (UID: \"086dab96-a305-4bcd-8d62-b36f73d429c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"
Jul 31 15:22:27.007375 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:27.007355    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/086dab96-a305-4bcd-8d62-b36f73d429c0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-758954f9f-w8hp6\" (UID: \"086dab96-a305-4bcd-8d62-b36f73d429c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"
Jul 31 15:22:27.007439 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:27.007424    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/086dab96-a305-4bcd-8d62-b36f73d429c0-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-758954f9f-w8hp6\" (UID: \"086dab96-a305-4bcd-8d62-b36f73d429c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"
Jul 31 15:22:27.009363 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:27.009340    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/086dab96-a305-4bcd-8d62-b36f73d429c0-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-758954f9f-w8hp6\" (UID: \"086dab96-a305-4bcd-8d62-b36f73d429c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"
Jul 31 15:22:27.018003 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:27.017982    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5bkn\" (UniqueName: \"kubernetes.io/projected/086dab96-a305-4bcd-8d62-b36f73d429c0-kube-api-access-s5bkn\") pod \"isvc-canary-rollback-v2-predictor-758954f9f-w8hp6\" (UID: \"086dab96-a305-4bcd-8d62-b36f73d429c0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"
Jul 31 15:22:27.088551 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:27.088515    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"
Jul 31 15:22:27.211399 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:27.211370    2559 generic.go:356] "Generic (PLEG): container finished" podID="872b0fc9-0302-4428-8784-a0a1e443db3f" containerID="5803bc37185849965f862c3a30d267556f57c348df5afa55c887e85cb047a440" exitCode=0
Jul 31 15:22:27.211577 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:27.211420    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628" event={"ID":"872b0fc9-0302-4428-8784-a0a1e443db3f","Type":"ContainerDied","Data":"5803bc37185849965f862c3a30d267556f57c348df5afa55c887e85cb047a440"}
Jul 31 15:22:27.211577 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:27.211476    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"
Jul 31 15:22:27.211577 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:27.211498    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628" event={"ID":"872b0fc9-0302-4428-8784-a0a1e443db3f","Type":"ContainerDied","Data":"4fd652c4427d86d0022075f42327eb53257e2e9e2482f7fbfb44c456087df2a9"}
Jul 31 15:22:27.211577 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:27.211539    2559 scope.go:117] "RemoveContainer" containerID="dddf5141e93cc050ef660baeb29d668f584572a5e91c19d4c4943f0bff32e0c9"
Jul 31 15:22:27.220373 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:27.220320    2559 scope.go:117] "RemoveContainer" containerID="5803bc37185849965f862c3a30d267556f57c348df5afa55c887e85cb047a440"
Jul 31 15:22:27.227716 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:27.227695    2559 scope.go:117] "RemoveContainer" containerID="e9888e500a4231256702617eee52c81432d9fdf932c0f93d39e0a92dbeb5707b"
Jul 31 15:22:27.235180 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:27.235162    2559 scope.go:117] "RemoveContainer" containerID="dddf5141e93cc050ef660baeb29d668f584572a5e91c19d4c4943f0bff32e0c9"
Jul 31 15:22:27.235495 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:22:27.235475    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dddf5141e93cc050ef660baeb29d668f584572a5e91c19d4c4943f0bff32e0c9\": container with ID starting with dddf5141e93cc050ef660baeb29d668f584572a5e91c19d4c4943f0bff32e0c9 not found: ID does not exist" containerID="dddf5141e93cc050ef660baeb29d668f584572a5e91c19d4c4943f0bff32e0c9"
Jul 31 15:22:27.235539 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:27.235505    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dddf5141e93cc050ef660baeb29d668f584572a5e91c19d4c4943f0bff32e0c9"} err="failed to get container status \"dddf5141e93cc050ef660baeb29d668f584572a5e91c19d4c4943f0bff32e0c9\": rpc error: code = NotFound desc = could not find container \"dddf5141e93cc050ef660baeb29d668f584572a5e91c19d4c4943f0bff32e0c9\": container with ID starting with dddf5141e93cc050ef660baeb29d668f584572a5e91c19d4c4943f0bff32e0c9 not found: ID does not exist"
Jul 31 15:22:27.235539 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:27.235524    2559 scope.go:117] "RemoveContainer" containerID="5803bc37185849965f862c3a30d267556f57c348df5afa55c887e85cb047a440"
Jul 31 15:22:27.235844 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:22:27.235822    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5803bc37185849965f862c3a30d267556f57c348df5afa55c887e85cb047a440\": container with ID starting with 5803bc37185849965f862c3a30d267556f57c348df5afa55c887e85cb047a440 not found: ID does not exist" containerID="5803bc37185849965f862c3a30d267556f57c348df5afa55c887e85cb047a440"
Jul 31 15:22:27.235936 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:27.235850    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5803bc37185849965f862c3a30d267556f57c348df5afa55c887e85cb047a440"} err="failed to get container status \"5803bc37185849965f862c3a30d267556f57c348df5afa55c887e85cb047a440\": rpc error: code = NotFound desc = could not find container \"5803bc37185849965f862c3a30d267556f57c348df5afa55c887e85cb047a440\": container with ID starting with 5803bc37185849965f862c3a30d267556f57c348df5afa55c887e85cb047a440 not found: ID does not exist"
Jul 31 15:22:27.235936 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:27.235870    2559 scope.go:117] "RemoveContainer" containerID="e9888e500a4231256702617eee52c81432d9fdf932c0f93d39e0a92dbeb5707b"
Jul 31 15:22:27.236179 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:22:27.236158    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9888e500a4231256702617eee52c81432d9fdf932c0f93d39e0a92dbeb5707b\": container with ID starting with e9888e500a4231256702617eee52c81432d9fdf932c0f93d39e0a92dbeb5707b not found: ID does not exist" containerID="e9888e500a4231256702617eee52c81432d9fdf932c0f93d39e0a92dbeb5707b"
Jul 31 15:22:27.236239 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:27.236189    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9888e500a4231256702617eee52c81432d9fdf932c0f93d39e0a92dbeb5707b"} err="failed to get container status \"e9888e500a4231256702617eee52c81432d9fdf932c0f93d39e0a92dbeb5707b\": rpc error: code = NotFound desc = could not find container \"e9888e500a4231256702617eee52c81432d9fdf932c0f93d39e0a92dbeb5707b\": container with ID starting with e9888e500a4231256702617eee52c81432d9fdf932c0f93d39e0a92dbeb5707b not found: ID does not exist"
Jul 31 15:22:27.236499 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:27.236483    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"]
Jul 31 15:22:27.238111 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:22:27.238091    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod086dab96_a305_4bcd_8d62_b36f73d429c0.slice/crio-269978c2936b89e2bbe2b2c07ed66de3c0ce088c7acaac5dd8a224e8b40d91ad WatchSource:0}: Error finding container 269978c2936b89e2bbe2b2c07ed66de3c0ce088c7acaac5dd8a224e8b40d91ad: Status 404 returned error can't find the container with id 269978c2936b89e2bbe2b2c07ed66de3c0ce088c7acaac5dd8a224e8b40d91ad
Jul 31 15:22:27.247811 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:27.247791    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"]
Jul 31 15:22:27.254987 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:27.254965    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-sk628"]
Jul 31 15:22:28.220644 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:28.220616    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="872b0fc9-0302-4428-8784-a0a1e443db3f" path="/var/lib/kubelet/pods/872b0fc9-0302-4428-8784-a0a1e443db3f/volumes"
Jul 31 15:22:28.221048 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:28.220964    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6" event={"ID":"086dab96-a305-4bcd-8d62-b36f73d429c0","Type":"ContainerStarted","Data":"9278bef7e481da55c01105793cf9f74de9b64a879f26f9e305cc4ef80c8ea09e"}
Jul 31 15:22:28.221048 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:28.221008    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6" event={"ID":"086dab96-a305-4bcd-8d62-b36f73d429c0","Type":"ContainerStarted","Data":"269978c2936b89e2bbe2b2c07ed66de3c0ce088c7acaac5dd8a224e8b40d91ad"}
Jul 31 15:22:31.231354 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:31.231319    2559 generic.go:356] "Generic (PLEG): container finished" podID="086dab96-a305-4bcd-8d62-b36f73d429c0" containerID="9278bef7e481da55c01105793cf9f74de9b64a879f26f9e305cc4ef80c8ea09e" exitCode=0
Jul 31 15:22:31.231763 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:31.231393    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6" event={"ID":"086dab96-a305-4bcd-8d62-b36f73d429c0","Type":"ContainerDied","Data":"9278bef7e481da55c01105793cf9f74de9b64a879f26f9e305cc4ef80c8ea09e"}
Jul 31 15:22:32.237829 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:32.237797    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6" event={"ID":"086dab96-a305-4bcd-8d62-b36f73d429c0","Type":"ContainerStarted","Data":"f7da2abd7fb39e4a211f1e62d408787c1176964bd8e4c0ce027be9868ec25408"}
Jul 31 15:22:32.238245 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:32.237859    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6" event={"ID":"086dab96-a305-4bcd-8d62-b36f73d429c0","Type":"ContainerStarted","Data":"f0b6ae00f84d63e4483993103bff693f1d752f3a0b64769ef56fecc8c8f4cdbd"}
Jul 31 15:22:32.238245 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:32.238057    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"
Jul 31 15:22:32.260050 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:32.259993    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6" podStartSLOduration=2.565805943 podStartE2EDuration="6.259975918s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.694169975s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:22:32.258654777 +0000 UTC m=+2048.570212400" watchObservedRunningTime="2026-07-31 15:22:32.259975918 +0000 UTC m=+2048.571533540"
Jul 31 15:22:33.241326 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:33.241292    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"
Jul 31 15:22:33.242649 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:33.242620    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6" podUID="086dab96-a305-4bcd-8d62-b36f73d429c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 31 15:22:34.244962 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:34.244918    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6" podUID="086dab96-a305-4bcd-8d62-b36f73d429c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 31 15:22:35.004255 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:35.004212    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq" podUID="77d625f7-0327-478a-b7e4-4334a02942d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 31 15:22:39.250590 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:39.250552    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"
Jul 31 15:22:39.251145 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:39.250948    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6" podUID="086dab96-a305-4bcd-8d62-b36f73d429c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 31 15:22:45.005403 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:45.005323    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"
Jul 31 15:22:49.251788 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:49.251752    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6" podUID="086dab96-a305-4bcd-8d62-b36f73d429c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 31 15:22:52.810867 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:52.810829    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"]
Jul 31 15:22:52.811260 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:52.811048    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq" podUID="77d625f7-0327-478a-b7e4-4334a02942d2" containerName="kserve-container" containerID="cri-o://c68da27eab27ac07f49aadbf05f22be3976d3c0c423bca5e33ad26e2629510bd" gracePeriod=30
Jul 31 15:22:52.811260 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:52.811146    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq" podUID="77d625f7-0327-478a-b7e4-4334a02942d2" containerName="kube-rbac-proxy" containerID="cri-o://cdf1f5f1391f4ff47798a28f0db54f062b2afce6c79698afd3e253e2fa144ad9" gracePeriod=30
Jul 31 15:22:53.313441 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:53.313410    2559 generic.go:356] "Generic (PLEG): container finished" podID="77d625f7-0327-478a-b7e4-4334a02942d2" containerID="cdf1f5f1391f4ff47798a28f0db54f062b2afce6c79698afd3e253e2fa144ad9" exitCode=2
Jul 31 15:22:53.313614 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:53.313467    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq" event={"ID":"77d625f7-0327-478a-b7e4-4334a02942d2","Type":"ContainerDied","Data":"cdf1f5f1391f4ff47798a28f0db54f062b2afce6c79698afd3e253e2fa144ad9"}
Jul 31 15:22:54.998310 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:54.998267    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq" podUID="77d625f7-0327-478a-b7e4-4334a02942d2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Jul 31 15:22:55.004975 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:55.004944    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq" podUID="77d625f7-0327-478a-b7e4-4334a02942d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 31 15:22:57.752050 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.752023    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"
Jul 31 15:22:57.851856 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.851821    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77d625f7-0327-478a-b7e4-4334a02942d2-kube-rbac-proxy-sar-config\") pod \"77d625f7-0327-478a-b7e4-4334a02942d2\" (UID: \"77d625f7-0327-478a-b7e4-4334a02942d2\") "
Jul 31 15:22:57.851856 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.851860    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rgqw\" (UniqueName: \"kubernetes.io/projected/77d625f7-0327-478a-b7e4-4334a02942d2-kube-api-access-9rgqw\") pod \"77d625f7-0327-478a-b7e4-4334a02942d2\" (UID: \"77d625f7-0327-478a-b7e4-4334a02942d2\") "
Jul 31 15:22:57.852172 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.851889    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77d625f7-0327-478a-b7e4-4334a02942d2-proxy-tls\") pod \"77d625f7-0327-478a-b7e4-4334a02942d2\" (UID: \"77d625f7-0327-478a-b7e4-4334a02942d2\") "
Jul 31 15:22:57.852172 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.851931    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77d625f7-0327-478a-b7e4-4334a02942d2-cabundle-cert\") pod \"77d625f7-0327-478a-b7e4-4334a02942d2\" (UID: \"77d625f7-0327-478a-b7e4-4334a02942d2\") "
Jul 31 15:22:57.852172 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.851964    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77d625f7-0327-478a-b7e4-4334a02942d2-kserve-provision-location\") pod \"77d625f7-0327-478a-b7e4-4334a02942d2\" (UID: \"77d625f7-0327-478a-b7e4-4334a02942d2\") "
Jul 31 15:22:57.852343 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.852260    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d625f7-0327-478a-b7e4-4334a02942d2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "77d625f7-0327-478a-b7e4-4334a02942d2" (UID: "77d625f7-0327-478a-b7e4-4334a02942d2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:22:57.852439 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.852389    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d625f7-0327-478a-b7e4-4334a02942d2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "77d625f7-0327-478a-b7e4-4334a02942d2" (UID: "77d625f7-0327-478a-b7e4-4334a02942d2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:22:57.852509 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.852458    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d625f7-0327-478a-b7e4-4334a02942d2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "77d625f7-0327-478a-b7e4-4334a02942d2" (UID: "77d625f7-0327-478a-b7e4-4334a02942d2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:22:57.854376 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.854342    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77d625f7-0327-478a-b7e4-4334a02942d2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "77d625f7-0327-478a-b7e4-4334a02942d2" (UID: "77d625f7-0327-478a-b7e4-4334a02942d2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:22:57.854376 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.854369    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d625f7-0327-478a-b7e4-4334a02942d2-kube-api-access-9rgqw" (OuterVolumeSpecName: "kube-api-access-9rgqw") pod "77d625f7-0327-478a-b7e4-4334a02942d2" (UID: "77d625f7-0327-478a-b7e4-4334a02942d2"). InnerVolumeSpecName "kube-api-access-9rgqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:22:57.953376 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.953345    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77d625f7-0327-478a-b7e4-4334a02942d2-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:22:57.953376 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.953373    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77d625f7-0327-478a-b7e4-4334a02942d2-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:22:57.953376 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.953383    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77d625f7-0327-478a-b7e4-4334a02942d2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:22:57.953625 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.953393    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rgqw\" (UniqueName: \"kubernetes.io/projected/77d625f7-0327-478a-b7e4-4334a02942d2-kube-api-access-9rgqw\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:22:57.953625 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.953403    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77d625f7-0327-478a-b7e4-4334a02942d2-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:22:57.984085 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.984042    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"]
Jul 31 15:22:57.984349 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.984334    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77d625f7-0327-478a-b7e4-4334a02942d2" containerName="storage-initializer"
Jul 31 15:22:57.984421 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.984350    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="77d625f7-0327-478a-b7e4-4334a02942d2" containerName="storage-initializer"
Jul 31 15:22:57.984421 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.984366    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77d625f7-0327-478a-b7e4-4334a02942d2" containerName="kube-rbac-proxy"
Jul 31 15:22:57.984421 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.984372    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="77d625f7-0327-478a-b7e4-4334a02942d2" containerName="kube-rbac-proxy"
Jul 31 15:22:57.984421 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.984389    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77d625f7-0327-478a-b7e4-4334a02942d2" containerName="kserve-container"
Jul 31 15:22:57.984421 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.984395    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="77d625f7-0327-478a-b7e4-4334a02942d2" containerName="kserve-container"
Jul 31 15:22:57.984656 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.984443    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="77d625f7-0327-478a-b7e4-4334a02942d2" containerName="kube-rbac-proxy"
Jul 31 15:22:57.984656 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.984456    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="77d625f7-0327-478a-b7e4-4334a02942d2" containerName="kserve-container"
Jul 31 15:22:57.987874 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.987851    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"
Jul 31 15:22:57.990173 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.990151    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Jul 31 15:22:57.990304 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.990275    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Jul 31 15:22:57.997368 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:57.997342    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"]
Jul 31 15:22:58.154871 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.154780    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-92rnj\" (UID: \"ca616dc3-0354-4ede-b007-3bd9d01ff7d9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"
Jul 31 15:22:58.154871 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.154826    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck24t\" (UniqueName: \"kubernetes.io/projected/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-kube-api-access-ck24t\") pod \"isvc-tensorflow-predictor-547dfc5669-92rnj\" (UID: \"ca616dc3-0354-4ede-b007-3bd9d01ff7d9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"
Jul 31 15:22:58.154871 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.154850    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-92rnj\" (UID: \"ca616dc3-0354-4ede-b007-3bd9d01ff7d9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"
Jul 31 15:22:58.155105 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.154887    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-92rnj\" (UID: \"ca616dc3-0354-4ede-b007-3bd9d01ff7d9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"
Jul 31 15:22:58.155105 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.154905    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-92rnj\" (UID: \"ca616dc3-0354-4ede-b007-3bd9d01ff7d9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"
Jul 31 15:22:58.256144 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.256106    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-92rnj\" (UID: \"ca616dc3-0354-4ede-b007-3bd9d01ff7d9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"
Jul 31 15:22:58.256144 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.256144    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-92rnj\" (UID: \"ca616dc3-0354-4ede-b007-3bd9d01ff7d9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"
Jul 31 15:22:58.256369 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.256177    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-92rnj\" (UID: \"ca616dc3-0354-4ede-b007-3bd9d01ff7d9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"
Jul 31 15:22:58.256369 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.256206    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck24t\" (UniqueName: \"kubernetes.io/projected/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-kube-api-access-ck24t\") pod \"isvc-tensorflow-predictor-547dfc5669-92rnj\" (UID: \"ca616dc3-0354-4ede-b007-3bd9d01ff7d9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"
Jul 31 15:22:58.256369 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.256226    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-92rnj\" (UID: \"ca616dc3-0354-4ede-b007-3bd9d01ff7d9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"
Jul 31 15:22:58.256604 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.256566    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-92rnj\" (UID: \"ca616dc3-0354-4ede-b007-3bd9d01ff7d9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"
Jul 31 15:22:58.256980 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.256960    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-92rnj\" (UID: \"ca616dc3-0354-4ede-b007-3bd9d01ff7d9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"
Jul 31 15:22:58.257048 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.256960    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-92rnj\" (UID: \"ca616dc3-0354-4ede-b007-3bd9d01ff7d9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"
Jul 31 15:22:58.258791 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.258770    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-92rnj\" (UID: \"ca616dc3-0354-4ede-b007-3bd9d01ff7d9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"
Jul 31 15:22:58.265563 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.265544    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck24t\" (UniqueName: \"kubernetes.io/projected/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-kube-api-access-ck24t\") pod \"isvc-tensorflow-predictor-547dfc5669-92rnj\" (UID: \"ca616dc3-0354-4ede-b007-3bd9d01ff7d9\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"
Jul 31 15:22:58.300285 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.300252    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"
Jul 31 15:22:58.333664 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.333634    2559 generic.go:356] "Generic (PLEG): container finished" podID="77d625f7-0327-478a-b7e4-4334a02942d2" containerID="c68da27eab27ac07f49aadbf05f22be3976d3c0c423bca5e33ad26e2629510bd" exitCode=0
Jul 31 15:22:58.333877 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.333695    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq" event={"ID":"77d625f7-0327-478a-b7e4-4334a02942d2","Type":"ContainerDied","Data":"c68da27eab27ac07f49aadbf05f22be3976d3c0c423bca5e33ad26e2629510bd"}
Jul 31 15:22:58.333877 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.333707    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"
Jul 31 15:22:58.333877 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.333776    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq" event={"ID":"77d625f7-0327-478a-b7e4-4334a02942d2","Type":"ContainerDied","Data":"3308bdbcf95f0e1004877e4beca4095c261db86bd780d9c5c249b584751fa425"}
Jul 31 15:22:58.333877 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.333814    2559 scope.go:117] "RemoveContainer" containerID="cdf1f5f1391f4ff47798a28f0db54f062b2afce6c79698afd3e253e2fa144ad9"
Jul 31 15:22:58.343483 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.343462    2559 scope.go:117] "RemoveContainer" containerID="c68da27eab27ac07f49aadbf05f22be3976d3c0c423bca5e33ad26e2629510bd"
Jul 31 15:22:58.353869 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.353847    2559 scope.go:117] "RemoveContainer" containerID="e3c95b305fa645320c11526b74a959128d31c54ed5bb35e780d5aaf298b271c8"
Jul 31 15:22:58.355073 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.355047    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"]
Jul 31 15:22:58.357302 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.357269    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7bf4c79bf4-txshq"]
Jul 31 15:22:58.370641 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.370616    2559 scope.go:117] "RemoveContainer" containerID="cdf1f5f1391f4ff47798a28f0db54f062b2afce6c79698afd3e253e2fa144ad9"
Jul 31 15:22:58.371063 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:22:58.371038    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdf1f5f1391f4ff47798a28f0db54f062b2afce6c79698afd3e253e2fa144ad9\": container with ID starting with cdf1f5f1391f4ff47798a28f0db54f062b2afce6c79698afd3e253e2fa144ad9 not found: ID does not exist" containerID="cdf1f5f1391f4ff47798a28f0db54f062b2afce6c79698afd3e253e2fa144ad9"
Jul 31 15:22:58.371152 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.371069    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf1f5f1391f4ff47798a28f0db54f062b2afce6c79698afd3e253e2fa144ad9"} err="failed to get container status \"cdf1f5f1391f4ff47798a28f0db54f062b2afce6c79698afd3e253e2fa144ad9\": rpc error: code = NotFound desc = could not find container \"cdf1f5f1391f4ff47798a28f0db54f062b2afce6c79698afd3e253e2fa144ad9\": container with ID starting with cdf1f5f1391f4ff47798a28f0db54f062b2afce6c79698afd3e253e2fa144ad9 not found: ID does not exist"
Jul 31 15:22:58.371152 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.371092    2559 scope.go:117] "RemoveContainer" containerID="c68da27eab27ac07f49aadbf05f22be3976d3c0c423bca5e33ad26e2629510bd"
Jul 31 15:22:58.371414 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:22:58.371368    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c68da27eab27ac07f49aadbf05f22be3976d3c0c423bca5e33ad26e2629510bd\": container with ID starting with c68da27eab27ac07f49aadbf05f22be3976d3c0c423bca5e33ad26e2629510bd not found: ID does not exist" containerID="c68da27eab27ac07f49aadbf05f22be3976d3c0c423bca5e33ad26e2629510bd"
Jul 31 15:22:58.371521 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.371410    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c68da27eab27ac07f49aadbf05f22be3976d3c0c423bca5e33ad26e2629510bd"} err="failed to get container status \"c68da27eab27ac07f49aadbf05f22be3976d3c0c423bca5e33ad26e2629510bd\": rpc error: code = NotFound desc = could not find container \"c68da27eab27ac07f49aadbf05f22be3976d3c0c423bca5e33ad26e2629510bd\": container with ID starting with c68da27eab27ac07f49aadbf05f22be3976d3c0c423bca5e33ad26e2629510bd not found: ID does not exist"
Jul 31 15:22:58.371521 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.371434    2559 scope.go:117] "RemoveContainer" containerID="e3c95b305fa645320c11526b74a959128d31c54ed5bb35e780d5aaf298b271c8"
Jul 31 15:22:58.371754 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:22:58.371718    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3c95b305fa645320c11526b74a959128d31c54ed5bb35e780d5aaf298b271c8\": container with ID starting with e3c95b305fa645320c11526b74a959128d31c54ed5bb35e780d5aaf298b271c8 not found: ID does not exist" containerID="e3c95b305fa645320c11526b74a959128d31c54ed5bb35e780d5aaf298b271c8"
Jul 31 15:22:58.371804 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.371761    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c95b305fa645320c11526b74a959128d31c54ed5bb35e780d5aaf298b271c8"} err="failed to get container status \"e3c95b305fa645320c11526b74a959128d31c54ed5bb35e780d5aaf298b271c8\": rpc error: code = NotFound desc = could not find container \"e3c95b305fa645320c11526b74a959128d31c54ed5bb35e780d5aaf298b271c8\": container with ID starting with e3c95b305fa645320c11526b74a959128d31c54ed5bb35e780d5aaf298b271c8 not found: ID does not exist"
Jul 31 15:22:58.443031 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:58.442943    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"]
Jul 31 15:22:58.446167 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:22:58.446140    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca616dc3_0354_4ede_b007_3bd9d01ff7d9.slice/crio-0f41f2b57996bead5c3b918895560abf0102707c788141ea1e851bd76e3da553 WatchSource:0}: Error finding container 0f41f2b57996bead5c3b918895560abf0102707c788141ea1e851bd76e3da553: Status 404 returned error can't find the container with id 0f41f2b57996bead5c3b918895560abf0102707c788141ea1e851bd76e3da553
Jul 31 15:22:59.251621 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:59.251580    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6" podUID="086dab96-a305-4bcd-8d62-b36f73d429c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 31 15:22:59.339797 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:59.339757    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj" event={"ID":"ca616dc3-0354-4ede-b007-3bd9d01ff7d9","Type":"ContainerStarted","Data":"8f991b0dd9302a561b51042e93847654f18629e25b4c04fd4cd15fb73c90b69c"}
Jul 31 15:22:59.340011 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:22:59.339862    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj" event={"ID":"ca616dc3-0354-4ede-b007-3bd9d01ff7d9","Type":"ContainerStarted","Data":"0f41f2b57996bead5c3b918895560abf0102707c788141ea1e851bd76e3da553"}
Jul 31 15:23:00.221269 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:00.221239    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77d625f7-0327-478a-b7e4-4334a02942d2" path="/var/lib/kubelet/pods/77d625f7-0327-478a-b7e4-4334a02942d2/volumes"
Jul 31 15:23:03.356216 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:03.356186    2559 generic.go:356] "Generic (PLEG): container finished" podID="ca616dc3-0354-4ede-b007-3bd9d01ff7d9" containerID="8f991b0dd9302a561b51042e93847654f18629e25b4c04fd4cd15fb73c90b69c" exitCode=0
Jul 31 15:23:03.356580 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:03.356251    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj" event={"ID":"ca616dc3-0354-4ede-b007-3bd9d01ff7d9","Type":"ContainerDied","Data":"8f991b0dd9302a561b51042e93847654f18629e25b4c04fd4cd15fb73c90b69c"}
Jul 31 15:23:03.357711 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:03.357695    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 15:23:07.376253 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:07.376215    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj" event={"ID":"ca616dc3-0354-4ede-b007-3bd9d01ff7d9","Type":"ContainerStarted","Data":"44d587c69de74c2ab09f7d1883314d2da62430a74a02cddd189d0c1d05cacb08"}
Jul 31 15:23:07.376620 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:07.376280    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj" event={"ID":"ca616dc3-0354-4ede-b007-3bd9d01ff7d9","Type":"ContainerStarted","Data":"19ff17cedeae231056c0792c41c0679465609753b15bff7efb349e1a253f295f"}
Jul 31 15:23:07.376620 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:07.376548    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"
Jul 31 15:23:07.376698 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:07.376680    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"
Jul 31 15:23:07.377920 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:07.377894    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj" podUID="ca616dc3-0354-4ede-b007-3bd9d01ff7d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 31 15:23:07.415066 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:07.415016    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj" podStartSLOduration=2.178656237 podStartE2EDuration="10.414999701s" totalImagesPullingTime="3.716394384s" totalInitContainerRuntime="4.51994908s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:23:07.414955556 +0000 UTC m=+2083.726513180" watchObservedRunningTime="2026-07-31 15:23:07.414999701 +0000 UTC m=+2083.726557322"
Jul 31 15:23:08.379938 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:08.379891    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj" podUID="ca616dc3-0354-4ede-b007-3bd9d01ff7d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 31 15:23:09.251806 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:09.251769    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6" podUID="086dab96-a305-4bcd-8d62-b36f73d429c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 31 15:23:13.384812 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:13.384785    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"
Jul 31 15:23:13.385312 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:13.385288    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj" podUID="ca616dc3-0354-4ede-b007-3bd9d01ff7d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 31 15:23:19.252047 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:19.252005    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6" podUID="086dab96-a305-4bcd-8d62-b36f73d429c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 31 15:23:23.385396 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:23.385368    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"
Jul 31 15:23:29.251397 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:29.251353    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6" podUID="086dab96-a305-4bcd-8d62-b36f73d429c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 31 15:23:39.251084 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:39.251040    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6" podUID="086dab96-a305-4bcd-8d62-b36f73d429c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 31 15:23:39.616492 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:39.616459    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"]
Jul 31 15:23:39.616782 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:39.616715    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj" podUID="ca616dc3-0354-4ede-b007-3bd9d01ff7d9" containerName="kserve-container" containerID="cri-o://19ff17cedeae231056c0792c41c0679465609753b15bff7efb349e1a253f295f" gracePeriod=30
Jul 31 15:23:39.616878 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:39.616790    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj" podUID="ca616dc3-0354-4ede-b007-3bd9d01ff7d9" containerName="kube-rbac-proxy" containerID="cri-o://44d587c69de74c2ab09f7d1883314d2da62430a74a02cddd189d0c1d05cacb08" gracePeriod=30
Jul 31 15:23:40.505228 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:40.505195    2559 generic.go:356] "Generic (PLEG): container finished" podID="ca616dc3-0354-4ede-b007-3bd9d01ff7d9" containerID="44d587c69de74c2ab09f7d1883314d2da62430a74a02cddd189d0c1d05cacb08" exitCode=2
Jul 31 15:23:40.505620 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:40.505243    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj" event={"ID":"ca616dc3-0354-4ede-b007-3bd9d01ff7d9","Type":"ContainerDied","Data":"44d587c69de74c2ab09f7d1883314d2da62430a74a02cddd189d0c1d05cacb08"}
Jul 31 15:23:43.380963 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:43.380927    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj" podUID="ca616dc3-0354-4ede-b007-3bd9d01ff7d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Jul 31 15:23:44.792751 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:44.792697    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"]
Jul 31 15:23:44.796119 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:44.796102    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"
Jul 31 15:23:44.798535 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:44.798515    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Jul 31 15:23:44.798650 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:44.798513    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Jul 31 15:23:44.809374 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:44.807956    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"]
Jul 31 15:23:44.898194 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:44.898150    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cd2f0c0-2807-4250-af78-317c03915c90-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v\" (UID: \"5cd2f0c0-2807-4250-af78-317c03915c90\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"
Jul 31 15:23:44.898370 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:44.898200    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cd2f0c0-2807-4250-af78-317c03915c90-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v\" (UID: \"5cd2f0c0-2807-4250-af78-317c03915c90\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"
Jul 31 15:23:44.898370 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:44.898287    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cd2f0c0-2807-4250-af78-317c03915c90-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v\" (UID: \"5cd2f0c0-2807-4250-af78-317c03915c90\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"
Jul 31 15:23:44.898370 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:44.898342    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpw99\" (UniqueName: \"kubernetes.io/projected/5cd2f0c0-2807-4250-af78-317c03915c90-kube-api-access-cpw99\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v\" (UID: \"5cd2f0c0-2807-4250-af78-317c03915c90\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"
Jul 31 15:23:44.898493 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:44.898384    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cd2f0c0-2807-4250-af78-317c03915c90-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v\" (UID: \"5cd2f0c0-2807-4250-af78-317c03915c90\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"
Jul 31 15:23:44.999650 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:44.999608    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cd2f0c0-2807-4250-af78-317c03915c90-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v\" (UID: \"5cd2f0c0-2807-4250-af78-317c03915c90\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"
Jul 31 15:23:44.999650 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:44.999658    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cd2f0c0-2807-4250-af78-317c03915c90-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v\" (UID: \"5cd2f0c0-2807-4250-af78-317c03915c90\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"
Jul 31 15:23:44.999952 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:44.999682    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cd2f0c0-2807-4250-af78-317c03915c90-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v\" (UID: \"5cd2f0c0-2807-4250-af78-317c03915c90\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"
Jul 31 15:23:44.999952 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:44.999710    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cd2f0c0-2807-4250-af78-317c03915c90-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v\" (UID: \"5cd2f0c0-2807-4250-af78-317c03915c90\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"
Jul 31 15:23:44.999952 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:44.999776    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpw99\" (UniqueName: \"kubernetes.io/projected/5cd2f0c0-2807-4250-af78-317c03915c90-kube-api-access-cpw99\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v\" (UID: \"5cd2f0c0-2807-4250-af78-317c03915c90\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"
Jul 31 15:23:44.999952 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:23:44.999808    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-serving-cert: secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Jul 31 15:23:44.999952 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:23:44.999876    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cd2f0c0-2807-4250-af78-317c03915c90-proxy-tls podName:5cd2f0c0-2807-4250-af78-317c03915c90 nodeName:}" failed. No retries permitted until 2026-07-31 15:23:45.499857168 +0000 UTC m=+2121.811414783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5cd2f0c0-2807-4250-af78-317c03915c90-proxy-tls") pod "isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v" (UID: "5cd2f0c0-2807-4250-af78-317c03915c90") : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Jul 31 15:23:45.000360 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:45.000192    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cd2f0c0-2807-4250-af78-317c03915c90-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v\" (UID: \"5cd2f0c0-2807-4250-af78-317c03915c90\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"
Jul 31 15:23:45.000435 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:45.000415    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cd2f0c0-2807-4250-af78-317c03915c90-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v\" (UID: \"5cd2f0c0-2807-4250-af78-317c03915c90\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"
Jul 31 15:23:45.000499 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:45.000477    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cd2f0c0-2807-4250-af78-317c03915c90-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v\" (UID: \"5cd2f0c0-2807-4250-af78-317c03915c90\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"
Jul 31 15:23:45.008990 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:45.008965    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpw99\" (UniqueName: \"kubernetes.io/projected/5cd2f0c0-2807-4250-af78-317c03915c90-kube-api-access-cpw99\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v\" (UID: \"5cd2f0c0-2807-4250-af78-317c03915c90\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"
Jul 31 15:23:45.504519 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:45.504476    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cd2f0c0-2807-4250-af78-317c03915c90-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v\" (UID: \"5cd2f0c0-2807-4250-af78-317c03915c90\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"
Jul 31 15:23:45.507178 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:45.507151    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cd2f0c0-2807-4250-af78-317c03915c90-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v\" (UID: \"5cd2f0c0-2807-4250-af78-317c03915c90\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"
Jul 31 15:23:45.705848 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:45.705797    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"
Jul 31 15:23:45.837589 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:45.837566    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"]
Jul 31 15:23:45.840205 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:23:45.840178    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd2f0c0_2807_4250_af78_317c03915c90.slice/crio-b71db33ca1a40d017f36bae2477b0b69e1dad7ea777735dc37295d5d40782afe WatchSource:0}: Error finding container b71db33ca1a40d017f36bae2477b0b69e1dad7ea777735dc37295d5d40782afe: Status 404 returned error can't find the container with id b71db33ca1a40d017f36bae2477b0b69e1dad7ea777735dc37295d5d40782afe
Jul 31 15:23:46.527922 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:46.527883    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v" event={"ID":"5cd2f0c0-2807-4250-af78-317c03915c90","Type":"ContainerStarted","Data":"dd33eebef9947d55b4c10a8e503ea1e8e39e16b6a710bf2fa8ded6f066e3e761"}
Jul 31 15:23:46.528070 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:46.527953    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v" event={"ID":"5cd2f0c0-2807-4250-af78-317c03915c90","Type":"ContainerStarted","Data":"b71db33ca1a40d017f36bae2477b0b69e1dad7ea777735dc37295d5d40782afe"}
Jul 31 15:23:48.380295 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:48.380250    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj" podUID="ca616dc3-0354-4ede-b007-3bd9d01ff7d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Jul 31 15:23:49.252563 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:49.252532    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"
Jul 31 15:23:50.545193 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:50.545108    2559 generic.go:356] "Generic (PLEG): container finished" podID="5cd2f0c0-2807-4250-af78-317c03915c90" containerID="dd33eebef9947d55b4c10a8e503ea1e8e39e16b6a710bf2fa8ded6f066e3e761" exitCode=0
Jul 31 15:23:50.545193 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:50.545173    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v" event={"ID":"5cd2f0c0-2807-4250-af78-317c03915c90","Type":"ContainerDied","Data":"dd33eebef9947d55b4c10a8e503ea1e8e39e16b6a710bf2fa8ded6f066e3e761"}
Jul 31 15:23:51.551498 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:51.551463    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v" event={"ID":"5cd2f0c0-2807-4250-af78-317c03915c90","Type":"ContainerStarted","Data":"864e88d88b3d4ca77ad436cbe39ce9b639f3fdb453e4a2e3f3345f7e94098d89"}
Jul 31 15:23:51.551918 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:51.551553    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v" event={"ID":"5cd2f0c0-2807-4250-af78-317c03915c90","Type":"ContainerStarted","Data":"3e20c70eff566b8aabf9d59e02df5e48cef2ca1c6edbc24f9e9a76e7d20a8a40"}
Jul 31 15:23:51.551918 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:51.551756    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"
Jul 31 15:23:51.571477 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:51.571426    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v" podStartSLOduration=3.251258944 podStartE2EDuration="7.571409723s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.320150779s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:23:51.570539852 +0000 UTC m=+2127.882097474" watchObservedRunningTime="2026-07-31 15:23:51.571409723 +0000 UTC m=+2127.882967344"
Jul 31 15:23:52.556050 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:52.556017    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"
Jul 31 15:23:52.557299 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:52.557275    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v" podUID="5cd2f0c0-2807-4250-af78-317c03915c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 31 15:23:53.381119 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:53.381082    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj" podUID="ca616dc3-0354-4ede-b007-3bd9d01ff7d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Jul 31 15:23:53.381291 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:53.381232    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"
Jul 31 15:23:53.560452 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:53.560412    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v" podUID="5cd2f0c0-2807-4250-af78-317c03915c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 31 15:23:56.654436 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:56.654402    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"]
Jul 31 15:23:56.655033 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:56.654690    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6" podUID="086dab96-a305-4bcd-8d62-b36f73d429c0" containerName="kserve-container" containerID="cri-o://f0b6ae00f84d63e4483993103bff693f1d752f3a0b64769ef56fecc8c8f4cdbd" gracePeriod=30
Jul 31 15:23:56.655033 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:56.654799    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6" podUID="086dab96-a305-4bcd-8d62-b36f73d429c0" containerName="kube-rbac-proxy" containerID="cri-o://f7da2abd7fb39e4a211f1e62d408787c1176964bd8e4c0ce027be9868ec25408" gracePeriod=30
Jul 31 15:23:56.698342 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:56.698300    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"]
Jul 31 15:23:56.702124 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:56.702102    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"
Jul 31 15:23:56.704934 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:56.704909    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 31 15:23:56.712597 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:56.712571    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"]
Jul 31 15:23:56.799654 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:56.799621    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cql85\" (UniqueName: \"kubernetes.io/projected/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-kube-api-access-cql85\") pod \"isvc-canary-rollback-predictor-7f985c86-smtwp\" (UID: \"7184eb72-63f2-4a88-ba7a-b5b5615b75f9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"
Jul 31 15:23:56.799654 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:56.799661    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f985c86-smtwp\" (UID: \"7184eb72-63f2-4a88-ba7a-b5b5615b75f9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"
Jul 31 15:23:56.799895 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:56.799682    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f985c86-smtwp\" (UID: \"7184eb72-63f2-4a88-ba7a-b5b5615b75f9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"
Jul 31 15:23:56.799895 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:56.799776    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f985c86-smtwp\" (UID: \"7184eb72-63f2-4a88-ba7a-b5b5615b75f9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"
Jul 31 15:23:56.799895 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:56.799822    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f985c86-smtwp\" (UID: \"7184eb72-63f2-4a88-ba7a-b5b5615b75f9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"
Jul 31 15:23:56.900989 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:56.900951    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f985c86-smtwp\" (UID: \"7184eb72-63f2-4a88-ba7a-b5b5615b75f9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"
Jul 31 15:23:56.901183 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:56.901017    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cql85\" (UniqueName: \"kubernetes.io/projected/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-kube-api-access-cql85\") pod \"isvc-canary-rollback-predictor-7f985c86-smtwp\" (UID: \"7184eb72-63f2-4a88-ba7a-b5b5615b75f9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"
Jul 31 15:23:56.901183 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:56.901041    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f985c86-smtwp\" (UID: \"7184eb72-63f2-4a88-ba7a-b5b5615b75f9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"
Jul 31 15:23:56.901183 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:56.901071    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f985c86-smtwp\" (UID: \"7184eb72-63f2-4a88-ba7a-b5b5615b75f9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"
Jul 31 15:23:56.901183 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:56.901094    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f985c86-smtwp\" (UID: \"7184eb72-63f2-4a88-ba7a-b5b5615b75f9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"
Jul 31 15:23:56.901541 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:56.901520    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f985c86-smtwp\" (UID: \"7184eb72-63f2-4a88-ba7a-b5b5615b75f9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"
Jul 31 15:23:56.901755 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:56.901712    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f985c86-smtwp\" (UID: \"7184eb72-63f2-4a88-ba7a-b5b5615b75f9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"
Jul 31 15:23:56.901813 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:56.901714    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f985c86-smtwp\" (UID: \"7184eb72-63f2-4a88-ba7a-b5b5615b75f9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"
Jul 31 15:23:56.903746 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:56.903706    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f985c86-smtwp\" (UID: \"7184eb72-63f2-4a88-ba7a-b5b5615b75f9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"
Jul 31 15:23:56.910805 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:56.910756    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cql85\" (UniqueName: \"kubernetes.io/projected/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-kube-api-access-cql85\") pod \"isvc-canary-rollback-predictor-7f985c86-smtwp\" (UID: \"7184eb72-63f2-4a88-ba7a-b5b5615b75f9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"
Jul 31 15:23:57.013036 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:57.012997    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"
Jul 31 15:23:57.152097 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:57.151912    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"]
Jul 31 15:23:57.154722 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:23:57.154695    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7184eb72_63f2_4a88_ba7a_b5b5615b75f9.slice/crio-a389a61c8c5417201e21837c4394c4694e7a4631f922b3c37aeac22dffa6674b WatchSource:0}: Error finding container a389a61c8c5417201e21837c4394c4694e7a4631f922b3c37aeac22dffa6674b: Status 404 returned error can't find the container with id a389a61c8c5417201e21837c4394c4694e7a4631f922b3c37aeac22dffa6674b
Jul 31 15:23:57.578328 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:57.578228    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp" event={"ID":"7184eb72-63f2-4a88-ba7a-b5b5615b75f9","Type":"ContainerStarted","Data":"ff8f334f66ef51dd26aaf1a049fe2700e971b0114946d88e07d3ea40b5d2546a"}
Jul 31 15:23:57.578328 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:57.578308    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp" event={"ID":"7184eb72-63f2-4a88-ba7a-b5b5615b75f9","Type":"ContainerStarted","Data":"a389a61c8c5417201e21837c4394c4694e7a4631f922b3c37aeac22dffa6674b"}
Jul 31 15:23:57.580391 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:57.580361    2559 generic.go:356] "Generic (PLEG): container finished" podID="086dab96-a305-4bcd-8d62-b36f73d429c0" containerID="f7da2abd7fb39e4a211f1e62d408787c1176964bd8e4c0ce027be9868ec25408" exitCode=2
Jul 31 15:23:57.580526 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:57.580434    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6" event={"ID":"086dab96-a305-4bcd-8d62-b36f73d429c0","Type":"ContainerDied","Data":"f7da2abd7fb39e4a211f1e62d408787c1176964bd8e4c0ce027be9868ec25408"}
Jul 31 15:23:58.380211 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:58.380169    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj" podUID="ca616dc3-0354-4ede-b007-3bd9d01ff7d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Jul 31 15:23:58.564748 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:58.564697    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"
Jul 31 15:23:58.565280 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:58.565254    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v" podUID="5cd2f0c0-2807-4250-af78-317c03915c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 31 15:23:59.245825 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:59.245779    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6" podUID="086dab96-a305-4bcd-8d62-b36f73d429c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Jul 31 15:23:59.251323 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:23:59.251299    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6" podUID="086dab96-a305-4bcd-8d62-b36f73d429c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 31 15:24:01.597674 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:01.597643    2559 generic.go:356] "Generic (PLEG): container finished" podID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" containerID="ff8f334f66ef51dd26aaf1a049fe2700e971b0114946d88e07d3ea40b5d2546a" exitCode=0
Jul 31 15:24:01.598083 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:01.597716    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp" event={"ID":"7184eb72-63f2-4a88-ba7a-b5b5615b75f9","Type":"ContainerDied","Data":"ff8f334f66ef51dd26aaf1a049fe2700e971b0114946d88e07d3ea40b5d2546a"}
Jul 31 15:24:01.898493 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:01.898465    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"
Jul 31 15:24:02.042244 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.042206    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/086dab96-a305-4bcd-8d62-b36f73d429c0-cabundle-cert\") pod \"086dab96-a305-4bcd-8d62-b36f73d429c0\" (UID: \"086dab96-a305-4bcd-8d62-b36f73d429c0\") "
Jul 31 15:24:02.042448 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.042252    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5bkn\" (UniqueName: \"kubernetes.io/projected/086dab96-a305-4bcd-8d62-b36f73d429c0-kube-api-access-s5bkn\") pod \"086dab96-a305-4bcd-8d62-b36f73d429c0\" (UID: \"086dab96-a305-4bcd-8d62-b36f73d429c0\") "
Jul 31 15:24:02.042448 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.042277    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/086dab96-a305-4bcd-8d62-b36f73d429c0-proxy-tls\") pod \"086dab96-a305-4bcd-8d62-b36f73d429c0\" (UID: \"086dab96-a305-4bcd-8d62-b36f73d429c0\") "
Jul 31 15:24:02.042448 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.042314    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/086dab96-a305-4bcd-8d62-b36f73d429c0-kube-rbac-proxy-sar-config\") pod \"086dab96-a305-4bcd-8d62-b36f73d429c0\" (UID: \"086dab96-a305-4bcd-8d62-b36f73d429c0\") "
Jul 31 15:24:02.042448 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.042358    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/086dab96-a305-4bcd-8d62-b36f73d429c0-kserve-provision-location\") pod \"086dab96-a305-4bcd-8d62-b36f73d429c0\" (UID: \"086dab96-a305-4bcd-8d62-b36f73d429c0\") "
Jul 31 15:24:02.042627 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.042530    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/086dab96-a305-4bcd-8d62-b36f73d429c0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "086dab96-a305-4bcd-8d62-b36f73d429c0" (UID: "086dab96-a305-4bcd-8d62-b36f73d429c0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:24:02.042693 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.042634    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/086dab96-a305-4bcd-8d62-b36f73d429c0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "086dab96-a305-4bcd-8d62-b36f73d429c0" (UID: "086dab96-a305-4bcd-8d62-b36f73d429c0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:24:02.042777 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.042698    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/086dab96-a305-4bcd-8d62-b36f73d429c0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "086dab96-a305-4bcd-8d62-b36f73d429c0" (UID: "086dab96-a305-4bcd-8d62-b36f73d429c0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:24:02.044545 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.044523    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/086dab96-a305-4bcd-8d62-b36f73d429c0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "086dab96-a305-4bcd-8d62-b36f73d429c0" (UID: "086dab96-a305-4bcd-8d62-b36f73d429c0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:24:02.044595 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.044545    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/086dab96-a305-4bcd-8d62-b36f73d429c0-kube-api-access-s5bkn" (OuterVolumeSpecName: "kube-api-access-s5bkn") pod "086dab96-a305-4bcd-8d62-b36f73d429c0" (UID: "086dab96-a305-4bcd-8d62-b36f73d429c0"). InnerVolumeSpecName "kube-api-access-s5bkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:24:02.143972 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.143877    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/086dab96-a305-4bcd-8d62-b36f73d429c0-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:24:02.143972 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.143923    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/086dab96-a305-4bcd-8d62-b36f73d429c0-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:24:02.143972 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.143934    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5bkn\" (UniqueName: \"kubernetes.io/projected/086dab96-a305-4bcd-8d62-b36f73d429c0-kube-api-access-s5bkn\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:24:02.143972 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.143944    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/086dab96-a305-4bcd-8d62-b36f73d429c0-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:24:02.143972 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.143952    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/086dab96-a305-4bcd-8d62-b36f73d429c0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:24:02.604272 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.604234    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp" event={"ID":"7184eb72-63f2-4a88-ba7a-b5b5615b75f9","Type":"ContainerStarted","Data":"1812bcf4c0a830cf6a51cb427508b3aa87b49681db142891a73c553b4d2e1be5"}
Jul 31 15:24:02.604787 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.604308    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp" event={"ID":"7184eb72-63f2-4a88-ba7a-b5b5615b75f9","Type":"ContainerStarted","Data":"ea11cdd693bc7d380c92ab2932e32b3bd7fe287c754a5ef9dce1c190ae8ebb5a"}
Jul 31 15:24:02.604787 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.604629    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"
Jul 31 15:24:02.604787 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.604766    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"
Jul 31 15:24:02.606218 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.606198    2559 generic.go:356] "Generic (PLEG): container finished" podID="086dab96-a305-4bcd-8d62-b36f73d429c0" containerID="f0b6ae00f84d63e4483993103bff693f1d752f3a0b64769ef56fecc8c8f4cdbd" exitCode=0
Jul 31 15:24:02.606336 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.606233    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6" event={"ID":"086dab96-a305-4bcd-8d62-b36f73d429c0","Type":"ContainerDied","Data":"f0b6ae00f84d63e4483993103bff693f1d752f3a0b64769ef56fecc8c8f4cdbd"}
Jul 31 15:24:02.606336 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.606267    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6" event={"ID":"086dab96-a305-4bcd-8d62-b36f73d429c0","Type":"ContainerDied","Data":"269978c2936b89e2bbe2b2c07ed66de3c0ce088c7acaac5dd8a224e8b40d91ad"}
Jul 31 15:24:02.606336 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.606272    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"
Jul 31 15:24:02.606336 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.606293    2559 scope.go:117] "RemoveContainer" containerID="f7da2abd7fb39e4a211f1e62d408787c1176964bd8e4c0ce027be9868ec25408"
Jul 31 15:24:02.606662 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.606280    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp" podUID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 31 15:24:02.614763 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.614716    2559 scope.go:117] "RemoveContainer" containerID="f0b6ae00f84d63e4483993103bff693f1d752f3a0b64769ef56fecc8c8f4cdbd"
Jul 31 15:24:02.622636 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.622619    2559 scope.go:117] "RemoveContainer" containerID="9278bef7e481da55c01105793cf9f74de9b64a879f26f9e305cc4ef80c8ea09e"
Jul 31 15:24:02.627129 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.626950    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp" podStartSLOduration=2.778983652 podStartE2EDuration="6.626932099s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.847948447s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:24:02.626246343 +0000 UTC m=+2138.937803965" watchObservedRunningTime="2026-07-31 15:24:02.626932099 +0000 UTC m=+2138.938489722"
Jul 31 15:24:02.631183 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.631163    2559 scope.go:117] "RemoveContainer" containerID="f7da2abd7fb39e4a211f1e62d408787c1176964bd8e4c0ce027be9868ec25408"
Jul 31 15:24:02.631444 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:24:02.631427    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7da2abd7fb39e4a211f1e62d408787c1176964bd8e4c0ce027be9868ec25408\": container with ID starting with f7da2abd7fb39e4a211f1e62d408787c1176964bd8e4c0ce027be9868ec25408 not found: ID does not exist" containerID="f7da2abd7fb39e4a211f1e62d408787c1176964bd8e4c0ce027be9868ec25408"
Jul 31 15:24:02.631494 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.631453    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7da2abd7fb39e4a211f1e62d408787c1176964bd8e4c0ce027be9868ec25408"} err="failed to get container status \"f7da2abd7fb39e4a211f1e62d408787c1176964bd8e4c0ce027be9868ec25408\": rpc error: code = NotFound desc = could not find container \"f7da2abd7fb39e4a211f1e62d408787c1176964bd8e4c0ce027be9868ec25408\": container with ID starting with f7da2abd7fb39e4a211f1e62d408787c1176964bd8e4c0ce027be9868ec25408 not found: ID does not exist"
Jul 31 15:24:02.631494 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.631470    2559 scope.go:117] "RemoveContainer" containerID="f0b6ae00f84d63e4483993103bff693f1d752f3a0b64769ef56fecc8c8f4cdbd"
Jul 31 15:24:02.631716 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:24:02.631693    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0b6ae00f84d63e4483993103bff693f1d752f3a0b64769ef56fecc8c8f4cdbd\": container with ID starting with f0b6ae00f84d63e4483993103bff693f1d752f3a0b64769ef56fecc8c8f4cdbd not found: ID does not exist" containerID="f0b6ae00f84d63e4483993103bff693f1d752f3a0b64769ef56fecc8c8f4cdbd"
Jul 31 15:24:02.631793 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.631725    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0b6ae00f84d63e4483993103bff693f1d752f3a0b64769ef56fecc8c8f4cdbd"} err="failed to get container status \"f0b6ae00f84d63e4483993103bff693f1d752f3a0b64769ef56fecc8c8f4cdbd\": rpc error: code = NotFound desc = could not find container \"f0b6ae00f84d63e4483993103bff693f1d752f3a0b64769ef56fecc8c8f4cdbd\": container with ID starting with f0b6ae00f84d63e4483993103bff693f1d752f3a0b64769ef56fecc8c8f4cdbd not found: ID does not exist"
Jul 31 15:24:02.631793 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.631765    2559 scope.go:117] "RemoveContainer" containerID="9278bef7e481da55c01105793cf9f74de9b64a879f26f9e305cc4ef80c8ea09e"
Jul 31 15:24:02.632065 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:24:02.632048    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9278bef7e481da55c01105793cf9f74de9b64a879f26f9e305cc4ef80c8ea09e\": container with ID starting with 9278bef7e481da55c01105793cf9f74de9b64a879f26f9e305cc4ef80c8ea09e not found: ID does not exist" containerID="9278bef7e481da55c01105793cf9f74de9b64a879f26f9e305cc4ef80c8ea09e"
Jul 31 15:24:02.632121 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.632068    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9278bef7e481da55c01105793cf9f74de9b64a879f26f9e305cc4ef80c8ea09e"} err="failed to get container status \"9278bef7e481da55c01105793cf9f74de9b64a879f26f9e305cc4ef80c8ea09e\": rpc error: code = NotFound desc = could not find container \"9278bef7e481da55c01105793cf9f74de9b64a879f26f9e305cc4ef80c8ea09e\": container with ID starting with 9278bef7e481da55c01105793cf9f74de9b64a879f26f9e305cc4ef80c8ea09e not found: ID does not exist"
Jul 31 15:24:02.640916 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.640890    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"]
Jul 31 15:24:02.647196 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:02.647174    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-758954f9f-w8hp6"]
Jul 31 15:24:03.380782 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:03.380723    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj" podUID="ca616dc3-0354-4ede-b007-3bd9d01ff7d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Jul 31 15:24:03.610831 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:03.610786    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp" podUID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 31 15:24:04.222384 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:04.222350    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="086dab96-a305-4bcd-8d62-b36f73d429c0" path="/var/lib/kubelet/pods/086dab96-a305-4bcd-8d62-b36f73d429c0/volumes"
Jul 31 15:24:08.380387 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:08.380345    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj" podUID="ca616dc3-0354-4ede-b007-3bd9d01ff7d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Jul 31 15:24:08.566054 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:08.566025    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"
Jul 31 15:24:08.615543 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:08.615519    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"
Jul 31 15:24:08.616024 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:08.616000    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp" podUID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 31 15:24:10.263002 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.262976    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"
Jul 31 15:24:10.410563 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.410462    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ck24t\" (UniqueName: \"kubernetes.io/projected/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-kube-api-access-ck24t\") pod \"ca616dc3-0354-4ede-b007-3bd9d01ff7d9\" (UID: \"ca616dc3-0354-4ede-b007-3bd9d01ff7d9\") "
Jul 31 15:24:10.410563 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.410518    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-cabundle-cert\") pod \"ca616dc3-0354-4ede-b007-3bd9d01ff7d9\" (UID: \"ca616dc3-0354-4ede-b007-3bd9d01ff7d9\") "
Jul 31 15:24:10.410563 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.410542    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-proxy-tls\") pod \"ca616dc3-0354-4ede-b007-3bd9d01ff7d9\" (UID: \"ca616dc3-0354-4ede-b007-3bd9d01ff7d9\") "
Jul 31 15:24:10.410888 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.410614    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-kserve-provision-location\") pod \"ca616dc3-0354-4ede-b007-3bd9d01ff7d9\" (UID: \"ca616dc3-0354-4ede-b007-3bd9d01ff7d9\") "
Jul 31 15:24:10.410888 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.410772    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-kube-rbac-proxy-sar-config\") pod \"ca616dc3-0354-4ede-b007-3bd9d01ff7d9\" (UID: \"ca616dc3-0354-4ede-b007-3bd9d01ff7d9\") "
Jul 31 15:24:10.411065 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.411041    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ca616dc3-0354-4ede-b007-3bd9d01ff7d9" (UID: "ca616dc3-0354-4ede-b007-3bd9d01ff7d9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:24:10.411119 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.411063    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ca616dc3-0354-4ede-b007-3bd9d01ff7d9" (UID: "ca616dc3-0354-4ede-b007-3bd9d01ff7d9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:24:10.412833 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.412808    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-kube-api-access-ck24t" (OuterVolumeSpecName: "kube-api-access-ck24t") pod "ca616dc3-0354-4ede-b007-3bd9d01ff7d9" (UID: "ca616dc3-0354-4ede-b007-3bd9d01ff7d9"). InnerVolumeSpecName "kube-api-access-ck24t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:24:10.412943 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.412869    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ca616dc3-0354-4ede-b007-3bd9d01ff7d9" (UID: "ca616dc3-0354-4ede-b007-3bd9d01ff7d9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:24:10.421972 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.421945    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ca616dc3-0354-4ede-b007-3bd9d01ff7d9" (UID: "ca616dc3-0354-4ede-b007-3bd9d01ff7d9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:24:10.511401 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.511368    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:24:10.511401 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.511392    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:24:10.511401 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.511402    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck24t\" (UniqueName: \"kubernetes.io/projected/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-kube-api-access-ck24t\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:24:10.511401 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.511411    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:24:10.511667 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.511420    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca616dc3-0354-4ede-b007-3bd9d01ff7d9-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:24:10.644887 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.644859    2559 generic.go:356] "Generic (PLEG): container finished" podID="ca616dc3-0354-4ede-b007-3bd9d01ff7d9" containerID="19ff17cedeae231056c0792c41c0679465609753b15bff7efb349e1a253f295f" exitCode=137
Jul 31 15:24:10.645066 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.644923    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj" event={"ID":"ca616dc3-0354-4ede-b007-3bd9d01ff7d9","Type":"ContainerDied","Data":"19ff17cedeae231056c0792c41c0679465609753b15bff7efb349e1a253f295f"}
Jul 31 15:24:10.645066 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.644923    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"
Jul 31 15:24:10.645066 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.644970    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj" event={"ID":"ca616dc3-0354-4ede-b007-3bd9d01ff7d9","Type":"ContainerDied","Data":"0f41f2b57996bead5c3b918895560abf0102707c788141ea1e851bd76e3da553"}
Jul 31 15:24:10.645066 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.644998    2559 scope.go:117] "RemoveContainer" containerID="44d587c69de74c2ab09f7d1883314d2da62430a74a02cddd189d0c1d05cacb08"
Jul 31 15:24:10.653239 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.653219    2559 scope.go:117] "RemoveContainer" containerID="19ff17cedeae231056c0792c41c0679465609753b15bff7efb349e1a253f295f"
Jul 31 15:24:10.660387 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.660370    2559 scope.go:117] "RemoveContainer" containerID="8f991b0dd9302a561b51042e93847654f18629e25b4c04fd4cd15fb73c90b69c"
Jul 31 15:24:10.667406 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.667391    2559 scope.go:117] "RemoveContainer" containerID="44d587c69de74c2ab09f7d1883314d2da62430a74a02cddd189d0c1d05cacb08"
Jul 31 15:24:10.667675 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:24:10.667658    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44d587c69de74c2ab09f7d1883314d2da62430a74a02cddd189d0c1d05cacb08\": container with ID starting with 44d587c69de74c2ab09f7d1883314d2da62430a74a02cddd189d0c1d05cacb08 not found: ID does not exist" containerID="44d587c69de74c2ab09f7d1883314d2da62430a74a02cddd189d0c1d05cacb08"
Jul 31 15:24:10.667746 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.667683    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44d587c69de74c2ab09f7d1883314d2da62430a74a02cddd189d0c1d05cacb08"} err="failed to get container status \"44d587c69de74c2ab09f7d1883314d2da62430a74a02cddd189d0c1d05cacb08\": rpc error: code = NotFound desc = could not find container \"44d587c69de74c2ab09f7d1883314d2da62430a74a02cddd189d0c1d05cacb08\": container with ID starting with 44d587c69de74c2ab09f7d1883314d2da62430a74a02cddd189d0c1d05cacb08 not found: ID does not exist"
Jul 31 15:24:10.667746 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.667700    2559 scope.go:117] "RemoveContainer" containerID="19ff17cedeae231056c0792c41c0679465609753b15bff7efb349e1a253f295f"
Jul 31 15:24:10.667921 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:24:10.667907    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19ff17cedeae231056c0792c41c0679465609753b15bff7efb349e1a253f295f\": container with ID starting with 19ff17cedeae231056c0792c41c0679465609753b15bff7efb349e1a253f295f not found: ID does not exist" containerID="19ff17cedeae231056c0792c41c0679465609753b15bff7efb349e1a253f295f"
Jul 31 15:24:10.667961 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.667924    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ff17cedeae231056c0792c41c0679465609753b15bff7efb349e1a253f295f"} err="failed to get container status \"19ff17cedeae231056c0792c41c0679465609753b15bff7efb349e1a253f295f\": rpc error: code = NotFound desc = could not find container \"19ff17cedeae231056c0792c41c0679465609753b15bff7efb349e1a253f295f\": container with ID starting with 19ff17cedeae231056c0792c41c0679465609753b15bff7efb349e1a253f295f not found: ID does not exist"
Jul 31 15:24:10.667961 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.667935    2559 scope.go:117] "RemoveContainer" containerID="8f991b0dd9302a561b51042e93847654f18629e25b4c04fd4cd15fb73c90b69c"
Jul 31 15:24:10.668162 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:24:10.668149    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f991b0dd9302a561b51042e93847654f18629e25b4c04fd4cd15fb73c90b69c\": container with ID starting with 8f991b0dd9302a561b51042e93847654f18629e25b4c04fd4cd15fb73c90b69c not found: ID does not exist" containerID="8f991b0dd9302a561b51042e93847654f18629e25b4c04fd4cd15fb73c90b69c"
Jul 31 15:24:10.668221 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.668164    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f991b0dd9302a561b51042e93847654f18629e25b4c04fd4cd15fb73c90b69c"} err="failed to get container status \"8f991b0dd9302a561b51042e93847654f18629e25b4c04fd4cd15fb73c90b69c\": rpc error: code = NotFound desc = could not find container \"8f991b0dd9302a561b51042e93847654f18629e25b4c04fd4cd15fb73c90b69c\": container with ID starting with 8f991b0dd9302a561b51042e93847654f18629e25b4c04fd4cd15fb73c90b69c not found: ID does not exist"
Jul 31 15:24:10.677921 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.677896    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"]
Jul 31 15:24:10.686154 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:10.686131    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-92rnj"]
Jul 31 15:24:12.220581 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:12.220547    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca616dc3-0354-4ede-b007-3bd9d01ff7d9" path="/var/lib/kubelet/pods/ca616dc3-0354-4ede-b007-3bd9d01ff7d9/volumes"
Jul 31 15:24:18.616023 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:18.615977    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp" podUID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 31 15:24:26.011019 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:26.010988    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"]
Jul 31 15:24:26.011447 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:26.011231    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v" podUID="5cd2f0c0-2807-4250-af78-317c03915c90" containerName="kserve-container" containerID="cri-o://3e20c70eff566b8aabf9d59e02df5e48cef2ca1c6edbc24f9e9a76e7d20a8a40" gracePeriod=30
Jul 31 15:24:26.011447 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:26.011264    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v" podUID="5cd2f0c0-2807-4250-af78-317c03915c90" containerName="kube-rbac-proxy" containerID="cri-o://864e88d88b3d4ca77ad436cbe39ce9b639f3fdb453e4a2e3f3345f7e94098d89" gracePeriod=30
Jul 31 15:24:26.711511 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:26.711481    2559 generic.go:356] "Generic (PLEG): container finished" podID="5cd2f0c0-2807-4250-af78-317c03915c90" containerID="864e88d88b3d4ca77ad436cbe39ce9b639f3fdb453e4a2e3f3345f7e94098d89" exitCode=2
Jul 31 15:24:26.711686 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:26.711558    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v" event={"ID":"5cd2f0c0-2807-4250-af78-317c03915c90","Type":"ContainerDied","Data":"864e88d88b3d4ca77ad436cbe39ce9b639f3fdb453e4a2e3f3345f7e94098d89"}
Jul 31 15:24:28.561155 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:28.561116    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v" podUID="5cd2f0c0-2807-4250-af78-317c03915c90" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Jul 31 15:24:28.616084 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:28.616044    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp" podUID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 31 15:24:31.174672 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.174627    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"]
Jul 31 15:24:31.175079 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.174965    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca616dc3-0354-4ede-b007-3bd9d01ff7d9" containerName="kube-rbac-proxy"
Jul 31 15:24:31.175079 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.174978    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca616dc3-0354-4ede-b007-3bd9d01ff7d9" containerName="kube-rbac-proxy"
Jul 31 15:24:31.175079 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.174996    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca616dc3-0354-4ede-b007-3bd9d01ff7d9" containerName="storage-initializer"
Jul 31 15:24:31.175079 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.175002    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca616dc3-0354-4ede-b007-3bd9d01ff7d9" containerName="storage-initializer"
Jul 31 15:24:31.175079 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.175011    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="086dab96-a305-4bcd-8d62-b36f73d429c0" containerName="kserve-container"
Jul 31 15:24:31.175079 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.175019    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="086dab96-a305-4bcd-8d62-b36f73d429c0" containerName="kserve-container"
Jul 31 15:24:31.175079 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.175027    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="086dab96-a305-4bcd-8d62-b36f73d429c0" containerName="storage-initializer"
Jul 31 15:24:31.175079 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.175032    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="086dab96-a305-4bcd-8d62-b36f73d429c0" containerName="storage-initializer"
Jul 31 15:24:31.175079 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.175042    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca616dc3-0354-4ede-b007-3bd9d01ff7d9" containerName="kserve-container"
Jul 31 15:24:31.175079 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.175047    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca616dc3-0354-4ede-b007-3bd9d01ff7d9" containerName="kserve-container"
Jul 31 15:24:31.175079 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.175054    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="086dab96-a305-4bcd-8d62-b36f73d429c0" containerName="kube-rbac-proxy"
Jul 31 15:24:31.175079 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.175064    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="086dab96-a305-4bcd-8d62-b36f73d429c0" containerName="kube-rbac-proxy"
Jul 31 15:24:31.175449 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.175136    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca616dc3-0354-4ede-b007-3bd9d01ff7d9" containerName="kserve-container"
Jul 31 15:24:31.175449 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.175149    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="086dab96-a305-4bcd-8d62-b36f73d429c0" containerName="kube-rbac-proxy"
Jul 31 15:24:31.175449 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.175161    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca616dc3-0354-4ede-b007-3bd9d01ff7d9" containerName="kube-rbac-proxy"
Jul 31 15:24:31.175449 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.175168    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="086dab96-a305-4bcd-8d62-b36f73d429c0" containerName="kserve-container"
Jul 31 15:24:31.177711 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.177690    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"
Jul 31 15:24:31.180122 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.180093    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Jul 31 15:24:31.180228 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.180176    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Jul 31 15:24:31.188188 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.188165    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"]
Jul 31 15:24:31.374222 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.374181    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c756461-2609-40fe-a9e4-985773ecc23f-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-47qmr\" (UID: \"9c756461-2609-40fe-a9e4-985773ecc23f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"
Jul 31 15:24:31.374222 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.374224    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c756461-2609-40fe-a9e4-985773ecc23f-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-47qmr\" (UID: \"9c756461-2609-40fe-a9e4-985773ecc23f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"
Jul 31 15:24:31.374459 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.374248    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c756461-2609-40fe-a9e4-985773ecc23f-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-47qmr\" (UID: \"9c756461-2609-40fe-a9e4-985773ecc23f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"
Jul 31 15:24:31.374459 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.374309    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c756461-2609-40fe-a9e4-985773ecc23f-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-47qmr\" (UID: \"9c756461-2609-40fe-a9e4-985773ecc23f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"
Jul 31 15:24:31.374459 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.374326    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv6v8\" (UniqueName: \"kubernetes.io/projected/9c756461-2609-40fe-a9e4-985773ecc23f-kube-api-access-vv6v8\") pod \"isvc-triton-predictor-5d7db58676-47qmr\" (UID: \"9c756461-2609-40fe-a9e4-985773ecc23f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"
Jul 31 15:24:31.475242 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.475154    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c756461-2609-40fe-a9e4-985773ecc23f-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-47qmr\" (UID: \"9c756461-2609-40fe-a9e4-985773ecc23f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"
Jul 31 15:24:31.475242 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.475189    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv6v8\" (UniqueName: \"kubernetes.io/projected/9c756461-2609-40fe-a9e4-985773ecc23f-kube-api-access-vv6v8\") pod \"isvc-triton-predictor-5d7db58676-47qmr\" (UID: \"9c756461-2609-40fe-a9e4-985773ecc23f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"
Jul 31 15:24:31.475242 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.475217    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c756461-2609-40fe-a9e4-985773ecc23f-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-47qmr\" (UID: \"9c756461-2609-40fe-a9e4-985773ecc23f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"
Jul 31 15:24:31.475242 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.475238    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c756461-2609-40fe-a9e4-985773ecc23f-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-47qmr\" (UID: \"9c756461-2609-40fe-a9e4-985773ecc23f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"
Jul 31 15:24:31.475605 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.475263    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c756461-2609-40fe-a9e4-985773ecc23f-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-47qmr\" (UID: \"9c756461-2609-40fe-a9e4-985773ecc23f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"
Jul 31 15:24:31.475670 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.475629    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c756461-2609-40fe-a9e4-985773ecc23f-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-47qmr\" (UID: \"9c756461-2609-40fe-a9e4-985773ecc23f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"
Jul 31 15:24:31.475903 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.475881    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c756461-2609-40fe-a9e4-985773ecc23f-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-47qmr\" (UID: \"9c756461-2609-40fe-a9e4-985773ecc23f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"
Jul 31 15:24:31.476031 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.475937    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c756461-2609-40fe-a9e4-985773ecc23f-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-47qmr\" (UID: \"9c756461-2609-40fe-a9e4-985773ecc23f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"
Jul 31 15:24:31.477865 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.477846    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c756461-2609-40fe-a9e4-985773ecc23f-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-47qmr\" (UID: \"9c756461-2609-40fe-a9e4-985773ecc23f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"
Jul 31 15:24:31.485379 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.485355    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv6v8\" (UniqueName: \"kubernetes.io/projected/9c756461-2609-40fe-a9e4-985773ecc23f-kube-api-access-vv6v8\") pod \"isvc-triton-predictor-5d7db58676-47qmr\" (UID: \"9c756461-2609-40fe-a9e4-985773ecc23f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"
Jul 31 15:24:31.488560 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.488543    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"
Jul 31 15:24:31.622190 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.622160    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"]
Jul 31 15:24:31.625035 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:24:31.625003    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c756461_2609_40fe_a9e4_985773ecc23f.slice/crio-31a2dfeb375ab079886c805bb3746bd101449251c6dde327cf04e1ba539f2b46 WatchSource:0}: Error finding container 31a2dfeb375ab079886c805bb3746bd101449251c6dde327cf04e1ba539f2b46: Status 404 returned error can't find the container with id 31a2dfeb375ab079886c805bb3746bd101449251c6dde327cf04e1ba539f2b46
Jul 31 15:24:31.731707 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.731609    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr" event={"ID":"9c756461-2609-40fe-a9e4-985773ecc23f","Type":"ContainerStarted","Data":"82103ca41ec62ae20663152b0a386a86200fa453a2aaafe9ae6a98c703ce7c08"}
Jul 31 15:24:31.731707 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:31.731672    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr" event={"ID":"9c756461-2609-40fe-a9e4-985773ecc23f","Type":"ContainerStarted","Data":"31a2dfeb375ab079886c805bb3746bd101449251c6dde327cf04e1ba539f2b46"}
Jul 31 15:24:33.560988 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:33.560945    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v" podUID="5cd2f0c0-2807-4250-af78-317c03915c90" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Jul 31 15:24:35.748161 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:35.748128    2559 generic.go:356] "Generic (PLEG): container finished" podID="9c756461-2609-40fe-a9e4-985773ecc23f" containerID="82103ca41ec62ae20663152b0a386a86200fa453a2aaafe9ae6a98c703ce7c08" exitCode=0
Jul 31 15:24:35.748538 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:35.748200    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr" event={"ID":"9c756461-2609-40fe-a9e4-985773ecc23f","Type":"ContainerDied","Data":"82103ca41ec62ae20663152b0a386a86200fa453a2aaafe9ae6a98c703ce7c08"}
Jul 31 15:24:38.561357 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:38.561309    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v" podUID="5cd2f0c0-2807-4250-af78-317c03915c90" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Jul 31 15:24:38.561883 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:38.561456    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"
Jul 31 15:24:38.616539 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:38.616150    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp" podUID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 31 15:24:43.561230 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:43.561179    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v" podUID="5cd2f0c0-2807-4250-af78-317c03915c90" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Jul 31 15:24:48.561585 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:48.561275    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v" podUID="5cd2f0c0-2807-4250-af78-317c03915c90" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Jul 31 15:24:48.616185 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:48.616146    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp" podUID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 31 15:24:53.561431 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:53.561383    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v" podUID="5cd2f0c0-2807-4250-af78-317c03915c90" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Jul 31 15:24:56.736402 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.736329    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"
Jul 31 15:24:56.797959 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.797307    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cd2f0c0-2807-4250-af78-317c03915c90-cabundle-cert\") pod \"5cd2f0c0-2807-4250-af78-317c03915c90\" (UID: \"5cd2f0c0-2807-4250-af78-317c03915c90\") "
Jul 31 15:24:56.797959 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.797364    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cd2f0c0-2807-4250-af78-317c03915c90-kube-rbac-proxy-sar-config\") pod \"5cd2f0c0-2807-4250-af78-317c03915c90\" (UID: \"5cd2f0c0-2807-4250-af78-317c03915c90\") "
Jul 31 15:24:56.797959 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.797413    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpw99\" (UniqueName: \"kubernetes.io/projected/5cd2f0c0-2807-4250-af78-317c03915c90-kube-api-access-cpw99\") pod \"5cd2f0c0-2807-4250-af78-317c03915c90\" (UID: \"5cd2f0c0-2807-4250-af78-317c03915c90\") "
Jul 31 15:24:56.797959 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.797460    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cd2f0c0-2807-4250-af78-317c03915c90-proxy-tls\") pod \"5cd2f0c0-2807-4250-af78-317c03915c90\" (UID: \"5cd2f0c0-2807-4250-af78-317c03915c90\") "
Jul 31 15:24:56.797959 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.797550    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cd2f0c0-2807-4250-af78-317c03915c90-kserve-provision-location\") pod \"5cd2f0c0-2807-4250-af78-317c03915c90\" (UID: \"5cd2f0c0-2807-4250-af78-317c03915c90\") "
Jul 31 15:24:56.797959 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.797800    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd2f0c0-2807-4250-af78-317c03915c90-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5cd2f0c0-2807-4250-af78-317c03915c90" (UID: "5cd2f0c0-2807-4250-af78-317c03915c90"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:24:56.797959 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.797814    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd2f0c0-2807-4250-af78-317c03915c90-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5cd2f0c0-2807-4250-af78-317c03915c90" (UID: "5cd2f0c0-2807-4250-af78-317c03915c90"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:24:56.802007 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.801656    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cd2f0c0-2807-4250-af78-317c03915c90-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5cd2f0c0-2807-4250-af78-317c03915c90" (UID: "5cd2f0c0-2807-4250-af78-317c03915c90"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:24:56.802574 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.802520    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd2f0c0-2807-4250-af78-317c03915c90-kube-api-access-cpw99" (OuterVolumeSpecName: "kube-api-access-cpw99") pod "5cd2f0c0-2807-4250-af78-317c03915c90" (UID: "5cd2f0c0-2807-4250-af78-317c03915c90"). InnerVolumeSpecName "kube-api-access-cpw99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:24:56.808368 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.808332    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd2f0c0-2807-4250-af78-317c03915c90-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5cd2f0c0-2807-4250-af78-317c03915c90" (UID: "5cd2f0c0-2807-4250-af78-317c03915c90"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:24:56.892075 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.892036    2559 generic.go:356] "Generic (PLEG): container finished" podID="5cd2f0c0-2807-4250-af78-317c03915c90" containerID="3e20c70eff566b8aabf9d59e02df5e48cef2ca1c6edbc24f9e9a76e7d20a8a40" exitCode=137
Jul 31 15:24:56.892262 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.892105    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"
Jul 31 15:24:56.892262 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.892122    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v" event={"ID":"5cd2f0c0-2807-4250-af78-317c03915c90","Type":"ContainerDied","Data":"3e20c70eff566b8aabf9d59e02df5e48cef2ca1c6edbc24f9e9a76e7d20a8a40"}
Jul 31 15:24:56.892262 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.892169    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v" event={"ID":"5cd2f0c0-2807-4250-af78-317c03915c90","Type":"ContainerDied","Data":"b71db33ca1a40d017f36bae2477b0b69e1dad7ea777735dc37295d5d40782afe"}
Jul 31 15:24:56.892262 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.892203    2559 scope.go:117] "RemoveContainer" containerID="864e88d88b3d4ca77ad436cbe39ce9b639f3fdb453e4a2e3f3345f7e94098d89"
Jul 31 15:24:56.898216 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.898126    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpw99\" (UniqueName: \"kubernetes.io/projected/5cd2f0c0-2807-4250-af78-317c03915c90-kube-api-access-cpw99\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:24:56.898216 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.898153    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cd2f0c0-2807-4250-af78-317c03915c90-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:24:56.898216 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.898167    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5cd2f0c0-2807-4250-af78-317c03915c90-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:24:56.898216 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.898181    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5cd2f0c0-2807-4250-af78-317c03915c90-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:24:56.898216 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.898194    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cd2f0c0-2807-4250-af78-317c03915c90-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:24:56.904027 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.903909    2559 scope.go:117] "RemoveContainer" containerID="3e20c70eff566b8aabf9d59e02df5e48cef2ca1c6edbc24f9e9a76e7d20a8a40"
Jul 31 15:24:56.915100 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.915055    2559 scope.go:117] "RemoveContainer" containerID="dd33eebef9947d55b4c10a8e503ea1e8e39e16b6a710bf2fa8ded6f066e3e761"
Jul 31 15:24:56.924247 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.924152    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"]
Jul 31 15:24:56.926554 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.926529    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-zxl8v"]
Jul 31 15:24:56.929270 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.929244    2559 scope.go:117] "RemoveContainer" containerID="864e88d88b3d4ca77ad436cbe39ce9b639f3fdb453e4a2e3f3345f7e94098d89"
Jul 31 15:24:56.929937 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:24:56.929895    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"864e88d88b3d4ca77ad436cbe39ce9b639f3fdb453e4a2e3f3345f7e94098d89\": container with ID starting with 864e88d88b3d4ca77ad436cbe39ce9b639f3fdb453e4a2e3f3345f7e94098d89 not found: ID does not exist" containerID="864e88d88b3d4ca77ad436cbe39ce9b639f3fdb453e4a2e3f3345f7e94098d89"
Jul 31 15:24:56.930039 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.929931    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"864e88d88b3d4ca77ad436cbe39ce9b639f3fdb453e4a2e3f3345f7e94098d89"} err="failed to get container status \"864e88d88b3d4ca77ad436cbe39ce9b639f3fdb453e4a2e3f3345f7e94098d89\": rpc error: code = NotFound desc = could not find container \"864e88d88b3d4ca77ad436cbe39ce9b639f3fdb453e4a2e3f3345f7e94098d89\": container with ID starting with 864e88d88b3d4ca77ad436cbe39ce9b639f3fdb453e4a2e3f3345f7e94098d89 not found: ID does not exist"
Jul 31 15:24:56.930039 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.929957    2559 scope.go:117] "RemoveContainer" containerID="3e20c70eff566b8aabf9d59e02df5e48cef2ca1c6edbc24f9e9a76e7d20a8a40"
Jul 31 15:24:56.930293 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:24:56.930269    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e20c70eff566b8aabf9d59e02df5e48cef2ca1c6edbc24f9e9a76e7d20a8a40\": container with ID starting with 3e20c70eff566b8aabf9d59e02df5e48cef2ca1c6edbc24f9e9a76e7d20a8a40 not found: ID does not exist" containerID="3e20c70eff566b8aabf9d59e02df5e48cef2ca1c6edbc24f9e9a76e7d20a8a40"
Jul 31 15:24:56.930365 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.930302    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e20c70eff566b8aabf9d59e02df5e48cef2ca1c6edbc24f9e9a76e7d20a8a40"} err="failed to get container status \"3e20c70eff566b8aabf9d59e02df5e48cef2ca1c6edbc24f9e9a76e7d20a8a40\": rpc error: code = NotFound desc = could not find container \"3e20c70eff566b8aabf9d59e02df5e48cef2ca1c6edbc24f9e9a76e7d20a8a40\": container with ID starting with 3e20c70eff566b8aabf9d59e02df5e48cef2ca1c6edbc24f9e9a76e7d20a8a40 not found: ID does not exist"
Jul 31 15:24:56.930365 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.930323    2559 scope.go:117] "RemoveContainer" containerID="dd33eebef9947d55b4c10a8e503ea1e8e39e16b6a710bf2fa8ded6f066e3e761"
Jul 31 15:24:56.930607 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:24:56.930579    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd33eebef9947d55b4c10a8e503ea1e8e39e16b6a710bf2fa8ded6f066e3e761\": container with ID starting with dd33eebef9947d55b4c10a8e503ea1e8e39e16b6a710bf2fa8ded6f066e3e761 not found: ID does not exist" containerID="dd33eebef9947d55b4c10a8e503ea1e8e39e16b6a710bf2fa8ded6f066e3e761"
Jul 31 15:24:56.930884 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:56.930854    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd33eebef9947d55b4c10a8e503ea1e8e39e16b6a710bf2fa8ded6f066e3e761"} err="failed to get container status \"dd33eebef9947d55b4c10a8e503ea1e8e39e16b6a710bf2fa8ded6f066e3e761\": rpc error: code = NotFound desc = could not find container \"dd33eebef9947d55b4c10a8e503ea1e8e39e16b6a710bf2fa8ded6f066e3e761\": container with ID starting with dd33eebef9947d55b4c10a8e503ea1e8e39e16b6a710bf2fa8ded6f066e3e761 not found: ID does not exist"
Jul 31 15:24:58.223123 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:58.223090    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cd2f0c0-2807-4250-af78-317c03915c90" path="/var/lib/kubelet/pods/5cd2f0c0-2807-4250-af78-317c03915c90/volumes"
Jul 31 15:24:58.616092 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:24:58.616050    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp" podUID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 31 15:25:08.616149 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:08.616108    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp" podUID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 31 15:25:18.617758 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:18.616724    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp" podUID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 31 15:25:22.216518 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:22.216333    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp" podUID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 31 15:25:32.216540 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:32.216485    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp" podUID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 31 15:25:42.222618 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:42.222584    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"
Jul 31 15:25:46.795548 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:46.794925    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"]
Jul 31 15:25:46.795548 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:46.795345    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp" podUID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" containerName="kserve-container" containerID="cri-o://ea11cdd693bc7d380c92ab2932e32b3bd7fe287c754a5ef9dce1c190ae8ebb5a" gracePeriod=30
Jul 31 15:25:46.799921 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:46.796300    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp" podUID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" containerName="kube-rbac-proxy" containerID="cri-o://1812bcf4c0a830cf6a51cb427508b3aa87b49681db142891a73c553b4d2e1be5" gracePeriod=30
Jul 31 15:25:48.611396 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:48.611346    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp" podUID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Jul 31 15:25:51.165217 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:51.165181    2559 generic.go:356] "Generic (PLEG): container finished" podID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" containerID="1812bcf4c0a830cf6a51cb427508b3aa87b49681db142891a73c553b4d2e1be5" exitCode=2
Jul 31 15:25:51.165698 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:51.165306    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp" event={"ID":"7184eb72-63f2-4a88-ba7a-b5b5615b75f9","Type":"ContainerDied","Data":"1812bcf4c0a830cf6a51cb427508b3aa87b49681db142891a73c553b4d2e1be5"}
Jul 31 15:25:52.014589 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.014558    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"]
Jul 31 15:25:52.014998 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.014982    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cd2f0c0-2807-4250-af78-317c03915c90" containerName="kube-rbac-proxy"
Jul 31 15:25:52.014998 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.015000    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cd2f0c0-2807-4250-af78-317c03915c90" containerName="kube-rbac-proxy"
Jul 31 15:25:52.015133 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.015019    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cd2f0c0-2807-4250-af78-317c03915c90" containerName="kserve-container"
Jul 31 15:25:52.015133 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.015030    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cd2f0c0-2807-4250-af78-317c03915c90" containerName="kserve-container"
Jul 31 15:25:52.015133 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.015041    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cd2f0c0-2807-4250-af78-317c03915c90" containerName="storage-initializer"
Jul 31 15:25:52.015133 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.015047    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cd2f0c0-2807-4250-af78-317c03915c90" containerName="storage-initializer"
Jul 31 15:25:52.015133 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.015111    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cd2f0c0-2807-4250-af78-317c03915c90" containerName="kserve-container"
Jul 31 15:25:52.015133 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.015119    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cd2f0c0-2807-4250-af78-317c03915c90" containerName="kube-rbac-proxy"
Jul 31 15:25:52.067829 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.067758    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"]
Jul 31 15:25:52.068028 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.067872    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"
Jul 31 15:25:52.070572 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.070551    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Jul 31 15:25:52.070666 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.070603    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 31 15:25:52.175838 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.175793    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c668a15-5b7c-4cec-8d02-403be047c243-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-87f57fc87-slb4x\" (UID: \"1c668a15-5b7c-4cec-8d02-403be047c243\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"
Jul 31 15:25:52.176282 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.175846    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c668a15-5b7c-4cec-8d02-403be047c243-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-87f57fc87-slb4x\" (UID: \"1c668a15-5b7c-4cec-8d02-403be047c243\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"
Jul 31 15:25:52.176282 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.175876    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c668a15-5b7c-4cec-8d02-403be047c243-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-87f57fc87-slb4x\" (UID: \"1c668a15-5b7c-4cec-8d02-403be047c243\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"
Jul 31 15:25:52.176282 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.175956    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c668a15-5b7c-4cec-8d02-403be047c243-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-87f57fc87-slb4x\" (UID: \"1c668a15-5b7c-4cec-8d02-403be047c243\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"
Jul 31 15:25:52.176282 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.175995    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzbkx\" (UniqueName: \"kubernetes.io/projected/1c668a15-5b7c-4cec-8d02-403be047c243-kube-api-access-gzbkx\") pod \"isvc-canary-stop-v2-predictor-87f57fc87-slb4x\" (UID: \"1c668a15-5b7c-4cec-8d02-403be047c243\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"
Jul 31 15:25:52.216414 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.216360    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp" podUID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 31 15:25:52.277335 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.277229    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c668a15-5b7c-4cec-8d02-403be047c243-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-87f57fc87-slb4x\" (UID: \"1c668a15-5b7c-4cec-8d02-403be047c243\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"
Jul 31 15:25:52.277335 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.277301    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c668a15-5b7c-4cec-8d02-403be047c243-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-87f57fc87-slb4x\" (UID: \"1c668a15-5b7c-4cec-8d02-403be047c243\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"
Jul 31 15:25:52.277564 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.277336    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c668a15-5b7c-4cec-8d02-403be047c243-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-87f57fc87-slb4x\" (UID: \"1c668a15-5b7c-4cec-8d02-403be047c243\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"
Jul 31 15:25:52.277564 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.277387    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c668a15-5b7c-4cec-8d02-403be047c243-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-87f57fc87-slb4x\" (UID: \"1c668a15-5b7c-4cec-8d02-403be047c243\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"
Jul 31 15:25:52.277564 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.277410    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzbkx\" (UniqueName: \"kubernetes.io/projected/1c668a15-5b7c-4cec-8d02-403be047c243-kube-api-access-gzbkx\") pod \"isvc-canary-stop-v2-predictor-87f57fc87-slb4x\" (UID: \"1c668a15-5b7c-4cec-8d02-403be047c243\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"
Jul 31 15:25:52.277832 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.277807    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c668a15-5b7c-4cec-8d02-403be047c243-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-87f57fc87-slb4x\" (UID: \"1c668a15-5b7c-4cec-8d02-403be047c243\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"
Jul 31 15:25:52.278038 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.277993    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c668a15-5b7c-4cec-8d02-403be047c243-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-87f57fc87-slb4x\" (UID: \"1c668a15-5b7c-4cec-8d02-403be047c243\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"
Jul 31 15:25:52.278266 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.278229    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c668a15-5b7c-4cec-8d02-403be047c243-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-87f57fc87-slb4x\" (UID: \"1c668a15-5b7c-4cec-8d02-403be047c243\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"
Jul 31 15:25:52.280541 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.280516    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c668a15-5b7c-4cec-8d02-403be047c243-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-87f57fc87-slb4x\" (UID: \"1c668a15-5b7c-4cec-8d02-403be047c243\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"
Jul 31 15:25:52.287020 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.286972    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzbkx\" (UniqueName: \"kubernetes.io/projected/1c668a15-5b7c-4cec-8d02-403be047c243-kube-api-access-gzbkx\") pod \"isvc-canary-stop-v2-predictor-87f57fc87-slb4x\" (UID: \"1c668a15-5b7c-4cec-8d02-403be047c243\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"
Jul 31 15:25:52.429573 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:52.429535    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"
Jul 31 15:25:53.176690 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:53.176640    2559 generic.go:356] "Generic (PLEG): container finished" podID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" containerID="ea11cdd693bc7d380c92ab2932e32b3bd7fe287c754a5ef9dce1c190ae8ebb5a" exitCode=0
Jul 31 15:25:53.177160 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:53.176718    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp" event={"ID":"7184eb72-63f2-4a88-ba7a-b5b5615b75f9","Type":"ContainerDied","Data":"ea11cdd693bc7d380c92ab2932e32b3bd7fe287c754a5ef9dce1c190ae8ebb5a"}
Jul 31 15:25:54.612290 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:54.612242    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp" podUID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": context deadline exceeded"
Jul 31 15:25:56.662798 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:56.662765    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"
Jul 31 15:25:56.714561 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:56.714516    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-cabundle-cert\") pod \"7184eb72-63f2-4a88-ba7a-b5b5615b75f9\" (UID: \"7184eb72-63f2-4a88-ba7a-b5b5615b75f9\") "
Jul 31 15:25:56.714865 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:56.714580    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-kube-rbac-proxy-sar-config\") pod \"7184eb72-63f2-4a88-ba7a-b5b5615b75f9\" (UID: \"7184eb72-63f2-4a88-ba7a-b5b5615b75f9\") "
Jul 31 15:25:56.714865 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:56.714609    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cql85\" (UniqueName: \"kubernetes.io/projected/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-kube-api-access-cql85\") pod \"7184eb72-63f2-4a88-ba7a-b5b5615b75f9\" (UID: \"7184eb72-63f2-4a88-ba7a-b5b5615b75f9\") "
Jul 31 15:25:56.714865 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:56.714636    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-proxy-tls\") pod \"7184eb72-63f2-4a88-ba7a-b5b5615b75f9\" (UID: \"7184eb72-63f2-4a88-ba7a-b5b5615b75f9\") "
Jul 31 15:25:56.714865 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:56.714757    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-kserve-provision-location\") pod \"7184eb72-63f2-4a88-ba7a-b5b5615b75f9\" (UID: \"7184eb72-63f2-4a88-ba7a-b5b5615b75f9\") "
Jul 31 15:25:56.715228 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:56.715201    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7184eb72-63f2-4a88-ba7a-b5b5615b75f9" (UID: "7184eb72-63f2-4a88-ba7a-b5b5615b75f9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:25:56.715300 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:56.715209    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7184eb72-63f2-4a88-ba7a-b5b5615b75f9" (UID: "7184eb72-63f2-4a88-ba7a-b5b5615b75f9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:25:56.715300 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:56.715276    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7184eb72-63f2-4a88-ba7a-b5b5615b75f9" (UID: "7184eb72-63f2-4a88-ba7a-b5b5615b75f9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:25:56.717527 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:56.717485    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7184eb72-63f2-4a88-ba7a-b5b5615b75f9" (UID: "7184eb72-63f2-4a88-ba7a-b5b5615b75f9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:25:56.717890 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:56.717860    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-kube-api-access-cql85" (OuterVolumeSpecName: "kube-api-access-cql85") pod "7184eb72-63f2-4a88-ba7a-b5b5615b75f9" (UID: "7184eb72-63f2-4a88-ba7a-b5b5615b75f9"). InnerVolumeSpecName "kube-api-access-cql85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:25:56.815926 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:56.815886    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:25:56.815926 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:56.815926    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:25:56.816130 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:56.815939    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:25:56.816130 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:56.815951    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cql85\" (UniqueName: \"kubernetes.io/projected/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-kube-api-access-cql85\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:25:56.816130 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:56.815962    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7184eb72-63f2-4a88-ba7a-b5b5615b75f9-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:25:57.194828 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:57.194795    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp" event={"ID":"7184eb72-63f2-4a88-ba7a-b5b5615b75f9","Type":"ContainerDied","Data":"a389a61c8c5417201e21837c4394c4694e7a4631f922b3c37aeac22dffa6674b"}
Jul 31 15:25:57.194828 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:57.194834    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"
Jul 31 15:25:57.195037 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:57.194862    2559 scope.go:117] "RemoveContainer" containerID="1812bcf4c0a830cf6a51cb427508b3aa87b49681db142891a73c553b4d2e1be5"
Jul 31 15:25:57.215761 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:57.215714    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"]
Jul 31 15:25:57.220055 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:57.220031    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-smtwp"]
Jul 31 15:25:57.229616 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:57.229596    2559 scope.go:117] "RemoveContainer" containerID="ea11cdd693bc7d380c92ab2932e32b3bd7fe287c754a5ef9dce1c190ae8ebb5a"
Jul 31 15:25:57.317615 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:57.317588    2559 scope.go:117] "RemoveContainer" containerID="ff8f334f66ef51dd26aaf1a049fe2700e971b0114946d88e07d3ea40b5d2546a"
Jul 31 15:25:57.447182 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:57.447105    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"]
Jul 31 15:25:58.220616 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:58.220585    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" path="/var/lib/kubelet/pods/7184eb72-63f2-4a88-ba7a-b5b5615b75f9/volumes"
Jul 31 15:25:58.285114 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:25:58.285066    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c668a15_5b7c_4cec_8d02_403be047c243.slice/crio-389a0ba8247399be9bdcaf89ec105c48ff994813822b147754829a13e2e67adb WatchSource:0}: Error finding container 389a0ba8247399be9bdcaf89ec105c48ff994813822b147754829a13e2e67adb: Status 404 returned error can't find the container with id 389a0ba8247399be9bdcaf89ec105c48ff994813822b147754829a13e2e67adb
Jul 31 15:25:59.206479 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:25:59.206442    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x" event={"ID":"1c668a15-5b7c-4cec-8d02-403be047c243","Type":"ContainerStarted","Data":"389a0ba8247399be9bdcaf89ec105c48ff994813822b147754829a13e2e67adb"}
Jul 31 15:26:31.338720 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:31.338621    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x" event={"ID":"1c668a15-5b7c-4cec-8d02-403be047c243","Type":"ContainerStarted","Data":"634d47be40b6d5b5244f249084f6b31e6dad43e59ea5375ece57f006d56d0284"}
Jul 31 15:26:32.345641 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:32.345601    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr" event={"ID":"9c756461-2609-40fe-a9e4-985773ecc23f","Type":"ContainerStarted","Data":"da384e42c1cb3711d2d7079f20f41d1a0715800bf6bba59eb6c7ef8c324d8e54"}
Jul 31 15:26:32.346163 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:32.345681    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr" event={"ID":"9c756461-2609-40fe-a9e4-985773ecc23f","Type":"ContainerStarted","Data":"368fb2e6eeb1b592e9bd5ac4b06eb33232aa99f9fdbc540081387b0691de30a4"}
Jul 31 15:26:32.367389 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:32.367331    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr" podStartSLOduration=1.458021983 podStartE2EDuration="2m1.367314138s" totalImagesPullingTime="1m55.967975989s" totalInitContainerRuntime="3.941316166s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:24:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:26:32.36522595 +0000 UTC m=+2288.676783585" watchObservedRunningTime="2026-07-31 15:26:32.367314138 +0000 UTC m=+2288.678871760"
Jul 31 15:26:33.352224 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:33.352186    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"
Jul 31 15:26:33.352705 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:33.352260    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"
Jul 31 15:26:33.353836 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:33.353808    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr" podUID="9c756461-2609-40fe-a9e4-985773ecc23f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 31 15:26:34.357375 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:34.357330    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr" podUID="9c756461-2609-40fe-a9e4-985773ecc23f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 31 15:26:35.362595 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:35.362559    2559 generic.go:356] "Generic (PLEG): container finished" podID="1c668a15-5b7c-4cec-8d02-403be047c243" containerID="634d47be40b6d5b5244f249084f6b31e6dad43e59ea5375ece57f006d56d0284" exitCode=0
Jul 31 15:26:35.363046 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:35.362626    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x" event={"ID":"1c668a15-5b7c-4cec-8d02-403be047c243","Type":"ContainerDied","Data":"634d47be40b6d5b5244f249084f6b31e6dad43e59ea5375ece57f006d56d0284"}
Jul 31 15:26:36.369229 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:36.369186    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x" event={"ID":"1c668a15-5b7c-4cec-8d02-403be047c243","Type":"ContainerStarted","Data":"74eb49695862485dc905be416050c8354b1401af7aacb2618e8d8c91f94a8b79"}
Jul 31 15:26:36.369758 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:36.369255    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x" event={"ID":"1c668a15-5b7c-4cec-8d02-403be047c243","Type":"ContainerStarted","Data":"82c080e3ab8373e8b97a78a9d501e717cd7d3c3d65a7d1902e5e2a391f470f5b"}
Jul 31 15:26:36.369758 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:36.369544    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"
Jul 31 15:26:36.369758 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:36.369748    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"
Jul 31 15:26:36.371364 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:36.371334    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x" podUID="1c668a15-5b7c-4cec-8d02-403be047c243" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 31 15:26:36.390901 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:36.390839    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x" podStartSLOduration=41.324861089 podStartE2EDuration="45.390820804s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.065959715s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:26:36.390148667 +0000 UTC m=+2292.701706289" watchObservedRunningTime="2026-07-31 15:26:36.390820804 +0000 UTC m=+2292.702378425"
Jul 31 15:26:37.373263 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:37.373217    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x" podUID="1c668a15-5b7c-4cec-8d02-403be047c243" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 31 15:26:39.362306 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:39.362275    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"
Jul 31 15:26:39.363012 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:39.362993    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"
Jul 31 15:26:42.377889 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:42.377860    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"
Jul 31 15:26:42.378403 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:42.378377    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x" podUID="1c668a15-5b7c-4cec-8d02-403be047c243" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 31 15:26:45.304305 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:45.304269    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"]
Jul 31 15:26:45.304825 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:45.304564    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr" podUID="9c756461-2609-40fe-a9e4-985773ecc23f" containerName="kube-rbac-proxy" containerID="cri-o://da384e42c1cb3711d2d7079f20f41d1a0715800bf6bba59eb6c7ef8c324d8e54" gracePeriod=30
Jul 31 15:26:45.304825 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:45.304539    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr" podUID="9c756461-2609-40fe-a9e4-985773ecc23f" containerName="kserve-container" containerID="cri-o://368fb2e6eeb1b592e9bd5ac4b06eb33232aa99f9fdbc540081387b0691de30a4" gracePeriod=30
Jul 31 15:26:46.407718 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:46.407684    2559 generic.go:356] "Generic (PLEG): container finished" podID="9c756461-2609-40fe-a9e4-985773ecc23f" containerID="da384e42c1cb3711d2d7079f20f41d1a0715800bf6bba59eb6c7ef8c324d8e54" exitCode=2
Jul 31 15:26:46.408119 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:46.407745    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr" event={"ID":"9c756461-2609-40fe-a9e4-985773ecc23f","Type":"ContainerDied","Data":"da384e42c1cb3711d2d7079f20f41d1a0715800bf6bba59eb6c7ef8c324d8e54"}
Jul 31 15:26:47.686329 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:47.686293    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"
Jul 31 15:26:47.750911 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:47.750807    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vv6v8\" (UniqueName: \"kubernetes.io/projected/9c756461-2609-40fe-a9e4-985773ecc23f-kube-api-access-vv6v8\") pod \"9c756461-2609-40fe-a9e4-985773ecc23f\" (UID: \"9c756461-2609-40fe-a9e4-985773ecc23f\") "
Jul 31 15:26:47.750911 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:47.750876    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c756461-2609-40fe-a9e4-985773ecc23f-kserve-provision-location\") pod \"9c756461-2609-40fe-a9e4-985773ecc23f\" (UID: \"9c756461-2609-40fe-a9e4-985773ecc23f\") "
Jul 31 15:26:47.750911 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:47.750908    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c756461-2609-40fe-a9e4-985773ecc23f-cabundle-cert\") pod \"9c756461-2609-40fe-a9e4-985773ecc23f\" (UID: \"9c756461-2609-40fe-a9e4-985773ecc23f\") "
Jul 31 15:26:47.751186 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:47.750935    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c756461-2609-40fe-a9e4-985773ecc23f-proxy-tls\") pod \"9c756461-2609-40fe-a9e4-985773ecc23f\" (UID: \"9c756461-2609-40fe-a9e4-985773ecc23f\") "
Jul 31 15:26:47.751186 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:47.750963    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c756461-2609-40fe-a9e4-985773ecc23f-kube-rbac-proxy-sar-config\") pod \"9c756461-2609-40fe-a9e4-985773ecc23f\" (UID: \"9c756461-2609-40fe-a9e4-985773ecc23f\") "
Jul 31 15:26:47.751307 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:47.751279    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c756461-2609-40fe-a9e4-985773ecc23f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9c756461-2609-40fe-a9e4-985773ecc23f" (UID: "9c756461-2609-40fe-a9e4-985773ecc23f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:26:47.751369 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:47.751314    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c756461-2609-40fe-a9e4-985773ecc23f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9c756461-2609-40fe-a9e4-985773ecc23f" (UID: "9c756461-2609-40fe-a9e4-985773ecc23f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:26:47.751419 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:47.751363    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c756461-2609-40fe-a9e4-985773ecc23f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9c756461-2609-40fe-a9e4-985773ecc23f" (UID: "9c756461-2609-40fe-a9e4-985773ecc23f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:26:47.753296 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:47.753266    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c756461-2609-40fe-a9e4-985773ecc23f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9c756461-2609-40fe-a9e4-985773ecc23f" (UID: "9c756461-2609-40fe-a9e4-985773ecc23f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:26:47.753397 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:47.753310    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c756461-2609-40fe-a9e4-985773ecc23f-kube-api-access-vv6v8" (OuterVolumeSpecName: "kube-api-access-vv6v8") pod "9c756461-2609-40fe-a9e4-985773ecc23f" (UID: "9c756461-2609-40fe-a9e4-985773ecc23f"). InnerVolumeSpecName "kube-api-access-vv6v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:26:47.852385 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:47.852349    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv6v8\" (UniqueName: \"kubernetes.io/projected/9c756461-2609-40fe-a9e4-985773ecc23f-kube-api-access-vv6v8\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:26:47.852385 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:47.852379    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c756461-2609-40fe-a9e4-985773ecc23f-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:26:47.852385 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:47.852389    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c756461-2609-40fe-a9e4-985773ecc23f-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:26:47.852385 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:47.852398    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c756461-2609-40fe-a9e4-985773ecc23f-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:26:47.852659 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:47.852406    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c756461-2609-40fe-a9e4-985773ecc23f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:26:48.416224 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:48.416191    2559 generic.go:356] "Generic (PLEG): container finished" podID="9c756461-2609-40fe-a9e4-985773ecc23f" containerID="368fb2e6eeb1b592e9bd5ac4b06eb33232aa99f9fdbc540081387b0691de30a4" exitCode=0
Jul 31 15:26:48.416439 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:48.416258    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"
Jul 31 15:26:48.416439 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:48.416272    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr" event={"ID":"9c756461-2609-40fe-a9e4-985773ecc23f","Type":"ContainerDied","Data":"368fb2e6eeb1b592e9bd5ac4b06eb33232aa99f9fdbc540081387b0691de30a4"}
Jul 31 15:26:48.416439 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:48.416336    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr" event={"ID":"9c756461-2609-40fe-a9e4-985773ecc23f","Type":"ContainerDied","Data":"31a2dfeb375ab079886c805bb3746bd101449251c6dde327cf04e1ba539f2b46"}
Jul 31 15:26:48.416439 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:48.416365    2559 scope.go:117] "RemoveContainer" containerID="da384e42c1cb3711d2d7079f20f41d1a0715800bf6bba59eb6c7ef8c324d8e54"
Jul 31 15:26:48.424893 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:48.424870    2559 scope.go:117] "RemoveContainer" containerID="368fb2e6eeb1b592e9bd5ac4b06eb33232aa99f9fdbc540081387b0691de30a4"
Jul 31 15:26:48.433105 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:48.433077    2559 scope.go:117] "RemoveContainer" containerID="82103ca41ec62ae20663152b0a386a86200fa453a2aaafe9ae6a98c703ce7c08"
Jul 31 15:26:48.441581 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:48.441557    2559 scope.go:117] "RemoveContainer" containerID="da384e42c1cb3711d2d7079f20f41d1a0715800bf6bba59eb6c7ef8c324d8e54"
Jul 31 15:26:48.441944 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:26:48.441922    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da384e42c1cb3711d2d7079f20f41d1a0715800bf6bba59eb6c7ef8c324d8e54\": container with ID starting with da384e42c1cb3711d2d7079f20f41d1a0715800bf6bba59eb6c7ef8c324d8e54 not found: ID does not exist" containerID="da384e42c1cb3711d2d7079f20f41d1a0715800bf6bba59eb6c7ef8c324d8e54"
Jul 31 15:26:48.442025 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:48.441956    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da384e42c1cb3711d2d7079f20f41d1a0715800bf6bba59eb6c7ef8c324d8e54"} err="failed to get container status \"da384e42c1cb3711d2d7079f20f41d1a0715800bf6bba59eb6c7ef8c324d8e54\": rpc error: code = NotFound desc = could not find container \"da384e42c1cb3711d2d7079f20f41d1a0715800bf6bba59eb6c7ef8c324d8e54\": container with ID starting with da384e42c1cb3711d2d7079f20f41d1a0715800bf6bba59eb6c7ef8c324d8e54 not found: ID does not exist"
Jul 31 15:26:48.442025 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:48.441977    2559 scope.go:117] "RemoveContainer" containerID="368fb2e6eeb1b592e9bd5ac4b06eb33232aa99f9fdbc540081387b0691de30a4"
Jul 31 15:26:48.442260 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:26:48.442245    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"368fb2e6eeb1b592e9bd5ac4b06eb33232aa99f9fdbc540081387b0691de30a4\": container with ID starting with 368fb2e6eeb1b592e9bd5ac4b06eb33232aa99f9fdbc540081387b0691de30a4 not found: ID does not exist" containerID="368fb2e6eeb1b592e9bd5ac4b06eb33232aa99f9fdbc540081387b0691de30a4"
Jul 31 15:26:48.442311 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:48.442266    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"368fb2e6eeb1b592e9bd5ac4b06eb33232aa99f9fdbc540081387b0691de30a4"} err="failed to get container status \"368fb2e6eeb1b592e9bd5ac4b06eb33232aa99f9fdbc540081387b0691de30a4\": rpc error: code = NotFound desc = could not find container \"368fb2e6eeb1b592e9bd5ac4b06eb33232aa99f9fdbc540081387b0691de30a4\": container with ID starting with 368fb2e6eeb1b592e9bd5ac4b06eb33232aa99f9fdbc540081387b0691de30a4 not found: ID does not exist"
Jul 31 15:26:48.442311 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:48.442280    2559 scope.go:117] "RemoveContainer" containerID="82103ca41ec62ae20663152b0a386a86200fa453a2aaafe9ae6a98c703ce7c08"
Jul 31 15:26:48.442457 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:48.442439    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"]
Jul 31 15:26:48.442549 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:26:48.442531    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82103ca41ec62ae20663152b0a386a86200fa453a2aaafe9ae6a98c703ce7c08\": container with ID starting with 82103ca41ec62ae20663152b0a386a86200fa453a2aaafe9ae6a98c703ce7c08 not found: ID does not exist" containerID="82103ca41ec62ae20663152b0a386a86200fa453a2aaafe9ae6a98c703ce7c08"
Jul 31 15:26:48.442588 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:48.442558    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82103ca41ec62ae20663152b0a386a86200fa453a2aaafe9ae6a98c703ce7c08"} err="failed to get container status \"82103ca41ec62ae20663152b0a386a86200fa453a2aaafe9ae6a98c703ce7c08\": rpc error: code = NotFound desc = could not find container \"82103ca41ec62ae20663152b0a386a86200fa453a2aaafe9ae6a98c703ce7c08\": container with ID starting with 82103ca41ec62ae20663152b0a386a86200fa453a2aaafe9ae6a98c703ce7c08 not found: ID does not exist"
Jul 31 15:26:48.449420 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:48.449398    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-47qmr"]
Jul 31 15:26:50.220934 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.220900    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c756461-2609-40fe-a9e4-985773ecc23f" path="/var/lib/kubelet/pods/9c756461-2609-40fe-a9e4-985773ecc23f/volumes"
Jul 31 15:26:50.511393 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.511363    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"]
Jul 31 15:26:50.511675 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.511662    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" containerName="storage-initializer"
Jul 31 15:26:50.511716 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.511678    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" containerName="storage-initializer"
Jul 31 15:26:50.511716 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.511689    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c756461-2609-40fe-a9e4-985773ecc23f" containerName="kserve-container"
Jul 31 15:26:50.511716 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.511696    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c756461-2609-40fe-a9e4-985773ecc23f" containerName="kserve-container"
Jul 31 15:26:50.511716 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.511703    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" containerName="kserve-container"
Jul 31 15:26:50.511716 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.511709    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" containerName="kserve-container"
Jul 31 15:26:50.511716 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.511717    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" containerName="kube-rbac-proxy"
Jul 31 15:26:50.511923 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.511722    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" containerName="kube-rbac-proxy"
Jul 31 15:26:50.511923 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.511753    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c756461-2609-40fe-a9e4-985773ecc23f" containerName="kube-rbac-proxy"
Jul 31 15:26:50.511923 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.511758    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c756461-2609-40fe-a9e4-985773ecc23f" containerName="kube-rbac-proxy"
Jul 31 15:26:50.511923 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.511769    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c756461-2609-40fe-a9e4-985773ecc23f" containerName="storage-initializer"
Jul 31 15:26:50.511923 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.511775    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c756461-2609-40fe-a9e4-985773ecc23f" containerName="storage-initializer"
Jul 31 15:26:50.511923 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.511833    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" containerName="kserve-container"
Jul 31 15:26:50.511923 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.511840    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7184eb72-63f2-4a88-ba7a-b5b5615b75f9" containerName="kube-rbac-proxy"
Jul 31 15:26:50.511923 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.511846    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c756461-2609-40fe-a9e4-985773ecc23f" containerName="kserve-container"
Jul 31 15:26:50.511923 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.511853    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c756461-2609-40fe-a9e4-985773ecc23f" containerName="kube-rbac-proxy"
Jul 31 15:26:50.550904 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.550862    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"]
Jul 31 15:26:50.551069 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.550994    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"
Jul 31 15:26:50.557580 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.557457    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Jul 31 15:26:50.558041 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.558023    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Jul 31 15:26:50.571243 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.571207    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36fbf3f8-8d7c-4167-9edc-0618abac7e96-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-zdpxg\" (UID: \"36fbf3f8-8d7c-4167-9edc-0618abac7e96\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"
Jul 31 15:26:50.571243 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.571241    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl655\" (UniqueName: \"kubernetes.io/projected/36fbf3f8-8d7c-4167-9edc-0618abac7e96-kube-api-access-wl655\") pod \"isvc-xgboost-predictor-7566d65bf7-zdpxg\" (UID: \"36fbf3f8-8d7c-4167-9edc-0618abac7e96\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"
Jul 31 15:26:50.571449 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.571269    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36fbf3f8-8d7c-4167-9edc-0618abac7e96-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-zdpxg\" (UID: \"36fbf3f8-8d7c-4167-9edc-0618abac7e96\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"
Jul 31 15:26:50.571449 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.571330    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36fbf3f8-8d7c-4167-9edc-0618abac7e96-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-zdpxg\" (UID: \"36fbf3f8-8d7c-4167-9edc-0618abac7e96\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"
Jul 31 15:26:50.571449 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.571363    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36fbf3f8-8d7c-4167-9edc-0618abac7e96-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-zdpxg\" (UID: \"36fbf3f8-8d7c-4167-9edc-0618abac7e96\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"
Jul 31 15:26:50.672454 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.672415    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36fbf3f8-8d7c-4167-9edc-0618abac7e96-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-zdpxg\" (UID: \"36fbf3f8-8d7c-4167-9edc-0618abac7e96\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"
Jul 31 15:26:50.672640 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.672459    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36fbf3f8-8d7c-4167-9edc-0618abac7e96-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-zdpxg\" (UID: \"36fbf3f8-8d7c-4167-9edc-0618abac7e96\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"
Jul 31 15:26:50.672640 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.672535    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36fbf3f8-8d7c-4167-9edc-0618abac7e96-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-zdpxg\" (UID: \"36fbf3f8-8d7c-4167-9edc-0618abac7e96\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"
Jul 31 15:26:50.672640 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.672581    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl655\" (UniqueName: \"kubernetes.io/projected/36fbf3f8-8d7c-4167-9edc-0618abac7e96-kube-api-access-wl655\") pod \"isvc-xgboost-predictor-7566d65bf7-zdpxg\" (UID: \"36fbf3f8-8d7c-4167-9edc-0618abac7e96\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"
Jul 31 15:26:50.672640 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.672606    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36fbf3f8-8d7c-4167-9edc-0618abac7e96-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-zdpxg\" (UID: \"36fbf3f8-8d7c-4167-9edc-0618abac7e96\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"
Jul 31 15:26:50.672986 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.672965    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36fbf3f8-8d7c-4167-9edc-0618abac7e96-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-zdpxg\" (UID: \"36fbf3f8-8d7c-4167-9edc-0618abac7e96\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"
Jul 31 15:26:50.673234 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.673208    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36fbf3f8-8d7c-4167-9edc-0618abac7e96-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-zdpxg\" (UID: \"36fbf3f8-8d7c-4167-9edc-0618abac7e96\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"
Jul 31 15:26:50.673347 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.673262    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36fbf3f8-8d7c-4167-9edc-0618abac7e96-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-zdpxg\" (UID: \"36fbf3f8-8d7c-4167-9edc-0618abac7e96\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"
Jul 31 15:26:50.675434 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.675411    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36fbf3f8-8d7c-4167-9edc-0618abac7e96-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-zdpxg\" (UID: \"36fbf3f8-8d7c-4167-9edc-0618abac7e96\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"
Jul 31 15:26:50.685608 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.685582    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl655\" (UniqueName: \"kubernetes.io/projected/36fbf3f8-8d7c-4167-9edc-0618abac7e96-kube-api-access-wl655\") pod \"isvc-xgboost-predictor-7566d65bf7-zdpxg\" (UID: \"36fbf3f8-8d7c-4167-9edc-0618abac7e96\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"
Jul 31 15:26:50.861622 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:50.861586    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"
Jul 31 15:26:51.019117 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:51.019091    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"]
Jul 31 15:26:51.021048 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:26:51.021013    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36fbf3f8_8d7c_4167_9edc_0618abac7e96.slice/crio-fc88d87bb692eaf6bc4c59bee804a4ab2c9f892b030ba8ff134ede863a49f80f WatchSource:0}: Error finding container fc88d87bb692eaf6bc4c59bee804a4ab2c9f892b030ba8ff134ede863a49f80f: Status 404 returned error can't find the container with id fc88d87bb692eaf6bc4c59bee804a4ab2c9f892b030ba8ff134ede863a49f80f
Jul 31 15:26:51.430538 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:51.430436    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg" event={"ID":"36fbf3f8-8d7c-4167-9edc-0618abac7e96","Type":"ContainerStarted","Data":"0e9ed0280e713783d16e54ee75552acb2072b00796db7ad3c146d1611b023679"}
Jul 31 15:26:51.430538 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:51.430495    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg" event={"ID":"36fbf3f8-8d7c-4167-9edc-0618abac7e96","Type":"ContainerStarted","Data":"fc88d87bb692eaf6bc4c59bee804a4ab2c9f892b030ba8ff134ede863a49f80f"}
Jul 31 15:26:52.379172 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:52.379123    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x" podUID="1c668a15-5b7c-4cec-8d02-403be047c243" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 31 15:26:55.447013 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:55.446978    2559 generic.go:356] "Generic (PLEG): container finished" podID="36fbf3f8-8d7c-4167-9edc-0618abac7e96" containerID="0e9ed0280e713783d16e54ee75552acb2072b00796db7ad3c146d1611b023679" exitCode=0
Jul 31 15:26:55.447425 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:26:55.447049    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg" event={"ID":"36fbf3f8-8d7c-4167-9edc-0618abac7e96","Type":"ContainerDied","Data":"0e9ed0280e713783d16e54ee75552acb2072b00796db7ad3c146d1611b023679"}
Jul 31 15:27:02.378753 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:27:02.378688    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x" podUID="1c668a15-5b7c-4cec-8d02-403be047c243" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 31 15:27:12.378811 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:27:12.378749    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x" podUID="1c668a15-5b7c-4cec-8d02-403be047c243" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 31 15:27:13.528462 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:27:13.528426    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg" event={"ID":"36fbf3f8-8d7c-4167-9edc-0618abac7e96","Type":"ContainerStarted","Data":"f4ee7dd2b3b7396146e3786e9d91e18be67fffa292f948c72a065373ef93967b"}
Jul 31 15:27:13.529003 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:27:13.528495    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg" event={"ID":"36fbf3f8-8d7c-4167-9edc-0618abac7e96","Type":"ContainerStarted","Data":"46dc09c5e3e534f59b38c638f9fc7efcbae4586bc9334f0d3022b7c3efb5bd74"}
Jul 31 15:27:13.529003 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:27:13.528689    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"
Jul 31 15:27:13.550795 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:27:13.550709    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg" podStartSLOduration=1.7240953509999999 podStartE2EDuration="23.550693732s" totalImagesPullingTime="17.88994422s" totalInitContainerRuntime="3.936654161s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:26:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:27:13.548464411 +0000 UTC m=+2329.860022036" watchObservedRunningTime="2026-07-31 15:27:13.550693732 +0000 UTC m=+2329.862251354"
Jul 31 15:27:14.531932 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:27:14.531897    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"
Jul 31 15:27:14.532904 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:27:14.532879    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg" podUID="36fbf3f8-8d7c-4167-9edc-0618abac7e96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 31 15:27:15.535896 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:27:15.535809    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg" podUID="36fbf3f8-8d7c-4167-9edc-0618abac7e96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 31 15:27:20.540742 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:27:20.540691    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"
Jul 31 15:27:20.541308 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:27:20.541281    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg" podUID="36fbf3f8-8d7c-4167-9edc-0618abac7e96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 31 15:27:22.379095 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:27:22.379055    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x" podUID="1c668a15-5b7c-4cec-8d02-403be047c243" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 31 15:27:30.541713 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:27:30.541672    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg" podUID="36fbf3f8-8d7c-4167-9edc-0618abac7e96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 31 15:27:32.378661 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:27:32.378607    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x" podUID="1c668a15-5b7c-4cec-8d02-403be047c243" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 31 15:27:40.541874 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:27:40.541828    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg" podUID="36fbf3f8-8d7c-4167-9edc-0618abac7e96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 31 15:27:42.379275 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:27:42.379235    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x" podUID="1c668a15-5b7c-4cec-8d02-403be047c243" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 31 15:27:50.541659 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:27:50.541619    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg" podUID="36fbf3f8-8d7c-4167-9edc-0618abac7e96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 31 15:27:52.378695 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:27:52.378658    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x" podUID="1c668a15-5b7c-4cec-8d02-403be047c243" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 31 15:27:53.216589 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:27:53.216546    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x" podUID="1c668a15-5b7c-4cec-8d02-403be047c243" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 31 15:28:00.541959 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:00.541916    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg" podUID="36fbf3f8-8d7c-4167-9edc-0618abac7e96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 31 15:28:03.217922 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:03.217889    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"
Jul 31 15:28:07.080649 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:07.080615    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"]
Jul 31 15:28:07.081084 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:07.080880    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x" podUID="1c668a15-5b7c-4cec-8d02-403be047c243" containerName="kserve-container" containerID="cri-o://82c080e3ab8373e8b97a78a9d501e717cd7d3c3d65a7d1902e5e2a391f470f5b" gracePeriod=30
Jul 31 15:28:07.081084 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:07.080936    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x" podUID="1c668a15-5b7c-4cec-8d02-403be047c243" containerName="kube-rbac-proxy" containerID="cri-o://74eb49695862485dc905be416050c8354b1401af7aacb2618e8d8c91f94a8b79" gracePeriod=30
Jul 31 15:28:07.374159 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:07.374114    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x" podUID="1c668a15-5b7c-4cec-8d02-403be047c243" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Jul 31 15:28:07.732587 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:07.732500    2559 generic.go:356] "Generic (PLEG): container finished" podID="1c668a15-5b7c-4cec-8d02-403be047c243" containerID="74eb49695862485dc905be416050c8354b1401af7aacb2618e8d8c91f94a8b79" exitCode=2
Jul 31 15:28:07.732587 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:07.732563    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x" event={"ID":"1c668a15-5b7c-4cec-8d02-403be047c243","Type":"ContainerDied","Data":"74eb49695862485dc905be416050c8354b1401af7aacb2618e8d8c91f94a8b79"}
Jul 31 15:28:10.541221 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:10.541181    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg" podUID="36fbf3f8-8d7c-4167-9edc-0618abac7e96" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 31 15:28:11.922322 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:11.922296    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"
Jul 31 15:28:12.035067 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.034987    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c668a15-5b7c-4cec-8d02-403be047c243-kube-rbac-proxy-sar-config\") pod \"1c668a15-5b7c-4cec-8d02-403be047c243\" (UID: \"1c668a15-5b7c-4cec-8d02-403be047c243\") "
Jul 31 15:28:12.035067 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.035043    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c668a15-5b7c-4cec-8d02-403be047c243-cabundle-cert\") pod \"1c668a15-5b7c-4cec-8d02-403be047c243\" (UID: \"1c668a15-5b7c-4cec-8d02-403be047c243\") "
Jul 31 15:28:12.035281 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.035141    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzbkx\" (UniqueName: \"kubernetes.io/projected/1c668a15-5b7c-4cec-8d02-403be047c243-kube-api-access-gzbkx\") pod \"1c668a15-5b7c-4cec-8d02-403be047c243\" (UID: \"1c668a15-5b7c-4cec-8d02-403be047c243\") "
Jul 31 15:28:12.035281 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.035181    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c668a15-5b7c-4cec-8d02-403be047c243-kserve-provision-location\") pod \"1c668a15-5b7c-4cec-8d02-403be047c243\" (UID: \"1c668a15-5b7c-4cec-8d02-403be047c243\") "
Jul 31 15:28:12.035281 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.035200    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c668a15-5b7c-4cec-8d02-403be047c243-proxy-tls\") pod \"1c668a15-5b7c-4cec-8d02-403be047c243\" (UID: \"1c668a15-5b7c-4cec-8d02-403be047c243\") "
Jul 31 15:28:12.035432 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.035376    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c668a15-5b7c-4cec-8d02-403be047c243-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1c668a15-5b7c-4cec-8d02-403be047c243" (UID: "1c668a15-5b7c-4cec-8d02-403be047c243"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:28:12.035491 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.035438    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c668a15-5b7c-4cec-8d02-403be047c243-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1c668a15-5b7c-4cec-8d02-403be047c243" (UID: "1c668a15-5b7c-4cec-8d02-403be047c243"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:28:12.035638 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.035616    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c668a15-5b7c-4cec-8d02-403be047c243-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1c668a15-5b7c-4cec-8d02-403be047c243" (UID: "1c668a15-5b7c-4cec-8d02-403be047c243"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:28:12.037470 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.037446    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c668a15-5b7c-4cec-8d02-403be047c243-kube-api-access-gzbkx" (OuterVolumeSpecName: "kube-api-access-gzbkx") pod "1c668a15-5b7c-4cec-8d02-403be047c243" (UID: "1c668a15-5b7c-4cec-8d02-403be047c243"). InnerVolumeSpecName "kube-api-access-gzbkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:28:12.037578 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.037560    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c668a15-5b7c-4cec-8d02-403be047c243-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1c668a15-5b7c-4cec-8d02-403be047c243" (UID: "1c668a15-5b7c-4cec-8d02-403be047c243"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:28:12.136311 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.136275    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c668a15-5b7c-4cec-8d02-403be047c243-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:28:12.136311 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.136304    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c668a15-5b7c-4cec-8d02-403be047c243-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:28:12.136311 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.136314    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzbkx\" (UniqueName: \"kubernetes.io/projected/1c668a15-5b7c-4cec-8d02-403be047c243-kube-api-access-gzbkx\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:28:12.136561 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.136326    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c668a15-5b7c-4cec-8d02-403be047c243-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:28:12.136561 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.136337    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c668a15-5b7c-4cec-8d02-403be047c243-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:28:12.752262 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.752230    2559 generic.go:356] "Generic (PLEG): container finished" podID="1c668a15-5b7c-4cec-8d02-403be047c243" containerID="82c080e3ab8373e8b97a78a9d501e717cd7d3c3d65a7d1902e5e2a391f470f5b" exitCode=0
Jul 31 15:28:12.752463 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.752288    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"
Jul 31 15:28:12.752463 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.752283    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x" event={"ID":"1c668a15-5b7c-4cec-8d02-403be047c243","Type":"ContainerDied","Data":"82c080e3ab8373e8b97a78a9d501e717cd7d3c3d65a7d1902e5e2a391f470f5b"}
Jul 31 15:28:12.752463 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.752425    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x" event={"ID":"1c668a15-5b7c-4cec-8d02-403be047c243","Type":"ContainerDied","Data":"389a0ba8247399be9bdcaf89ec105c48ff994813822b147754829a13e2e67adb"}
Jul 31 15:28:12.752463 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.752454    2559 scope.go:117] "RemoveContainer" containerID="74eb49695862485dc905be416050c8354b1401af7aacb2618e8d8c91f94a8b79"
Jul 31 15:28:12.760579 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.760561    2559 scope.go:117] "RemoveContainer" containerID="82c080e3ab8373e8b97a78a9d501e717cd7d3c3d65a7d1902e5e2a391f470f5b"
Jul 31 15:28:12.767783 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.767759    2559 scope.go:117] "RemoveContainer" containerID="634d47be40b6d5b5244f249084f6b31e6dad43e59ea5375ece57f006d56d0284"
Jul 31 15:28:12.770157 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.770137    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"]
Jul 31 15:28:12.776392 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.776371    2559 scope.go:117] "RemoveContainer" containerID="74eb49695862485dc905be416050c8354b1401af7aacb2618e8d8c91f94a8b79"
Jul 31 15:28:12.776625 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.776602    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-87f57fc87-slb4x"]
Jul 31 15:28:12.776765 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:28:12.776688    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74eb49695862485dc905be416050c8354b1401af7aacb2618e8d8c91f94a8b79\": container with ID starting with 74eb49695862485dc905be416050c8354b1401af7aacb2618e8d8c91f94a8b79 not found: ID does not exist" containerID="74eb49695862485dc905be416050c8354b1401af7aacb2618e8d8c91f94a8b79"
Jul 31 15:28:12.776845 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.776720    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74eb49695862485dc905be416050c8354b1401af7aacb2618e8d8c91f94a8b79"} err="failed to get container status \"74eb49695862485dc905be416050c8354b1401af7aacb2618e8d8c91f94a8b79\": rpc error: code = NotFound desc = could not find container \"74eb49695862485dc905be416050c8354b1401af7aacb2618e8d8c91f94a8b79\": container with ID starting with 74eb49695862485dc905be416050c8354b1401af7aacb2618e8d8c91f94a8b79 not found: ID does not exist"
Jul 31 15:28:12.776845 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.776786    2559 scope.go:117] "RemoveContainer" containerID="82c080e3ab8373e8b97a78a9d501e717cd7d3c3d65a7d1902e5e2a391f470f5b"
Jul 31 15:28:12.777062 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:28:12.777044    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82c080e3ab8373e8b97a78a9d501e717cd7d3c3d65a7d1902e5e2a391f470f5b\": container with ID starting with 82c080e3ab8373e8b97a78a9d501e717cd7d3c3d65a7d1902e5e2a391f470f5b not found: ID does not exist" containerID="82c080e3ab8373e8b97a78a9d501e717cd7d3c3d65a7d1902e5e2a391f470f5b"
Jul 31 15:28:12.777127 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.777070    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82c080e3ab8373e8b97a78a9d501e717cd7d3c3d65a7d1902e5e2a391f470f5b"} err="failed to get container status \"82c080e3ab8373e8b97a78a9d501e717cd7d3c3d65a7d1902e5e2a391f470f5b\": rpc error: code = NotFound desc = could not find container \"82c080e3ab8373e8b97a78a9d501e717cd7d3c3d65a7d1902e5e2a391f470f5b\": container with ID starting with 82c080e3ab8373e8b97a78a9d501e717cd7d3c3d65a7d1902e5e2a391f470f5b not found: ID does not exist"
Jul 31 15:28:12.777127 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.777090    2559 scope.go:117] "RemoveContainer" containerID="634d47be40b6d5b5244f249084f6b31e6dad43e59ea5375ece57f006d56d0284"
Jul 31 15:28:12.777340 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:28:12.777324    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"634d47be40b6d5b5244f249084f6b31e6dad43e59ea5375ece57f006d56d0284\": container with ID starting with 634d47be40b6d5b5244f249084f6b31e6dad43e59ea5375ece57f006d56d0284 not found: ID does not exist" containerID="634d47be40b6d5b5244f249084f6b31e6dad43e59ea5375ece57f006d56d0284"
Jul 31 15:28:12.777380 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:12.777348    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"634d47be40b6d5b5244f249084f6b31e6dad43e59ea5375ece57f006d56d0284"} err="failed to get container status \"634d47be40b6d5b5244f249084f6b31e6dad43e59ea5375ece57f006d56d0284\": rpc error: code = NotFound desc = could not find container \"634d47be40b6d5b5244f249084f6b31e6dad43e59ea5375ece57f006d56d0284\": container with ID starting with 634d47be40b6d5b5244f249084f6b31e6dad43e59ea5375ece57f006d56d0284 not found: ID does not exist"
Jul 31 15:28:14.221064 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:14.221025    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c668a15-5b7c-4cec-8d02-403be047c243" path="/var/lib/kubelet/pods/1c668a15-5b7c-4cec-8d02-403be047c243/volumes"
Jul 31 15:28:20.541987 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:20.541956    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"
Jul 31 15:28:30.592930 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:30.592897    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"]
Jul 31 15:28:30.593463 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:30.593140    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg" podUID="36fbf3f8-8d7c-4167-9edc-0618abac7e96" containerName="kserve-container" containerID="cri-o://46dc09c5e3e534f59b38c638f9fc7efcbae4586bc9334f0d3022b7c3efb5bd74" gracePeriod=30
Jul 31 15:28:30.593463 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:30.593205    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg" podUID="36fbf3f8-8d7c-4167-9edc-0618abac7e96" containerName="kube-rbac-proxy" containerID="cri-o://f4ee7dd2b3b7396146e3786e9d91e18be67fffa292f948c72a065373ef93967b" gracePeriod=30
Jul 31 15:28:30.828489 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:30.828455    2559 generic.go:356] "Generic (PLEG): container finished" podID="36fbf3f8-8d7c-4167-9edc-0618abac7e96" containerID="f4ee7dd2b3b7396146e3786e9d91e18be67fffa292f948c72a065373ef93967b" exitCode=2
Jul 31 15:28:30.828665 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:30.828528    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg" event={"ID":"36fbf3f8-8d7c-4167-9edc-0618abac7e96","Type":"ContainerDied","Data":"f4ee7dd2b3b7396146e3786e9d91e18be67fffa292f948c72a065373ef93967b"}
Jul 31 15:28:34.337024 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.336999    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"
Jul 31 15:28:34.409401 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.409368    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wl655\" (UniqueName: \"kubernetes.io/projected/36fbf3f8-8d7c-4167-9edc-0618abac7e96-kube-api-access-wl655\") pod \"36fbf3f8-8d7c-4167-9edc-0618abac7e96\" (UID: \"36fbf3f8-8d7c-4167-9edc-0618abac7e96\") "
Jul 31 15:28:34.409600 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.409426    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36fbf3f8-8d7c-4167-9edc-0618abac7e96-kube-rbac-proxy-sar-config\") pod \"36fbf3f8-8d7c-4167-9edc-0618abac7e96\" (UID: \"36fbf3f8-8d7c-4167-9edc-0618abac7e96\") "
Jul 31 15:28:34.409600 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.409458    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36fbf3f8-8d7c-4167-9edc-0618abac7e96-cabundle-cert\") pod \"36fbf3f8-8d7c-4167-9edc-0618abac7e96\" (UID: \"36fbf3f8-8d7c-4167-9edc-0618abac7e96\") "
Jul 31 15:28:34.409753 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.409621    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36fbf3f8-8d7c-4167-9edc-0618abac7e96-proxy-tls\") pod \"36fbf3f8-8d7c-4167-9edc-0618abac7e96\" (UID: \"36fbf3f8-8d7c-4167-9edc-0618abac7e96\") "
Jul 31 15:28:34.409753 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.409675    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36fbf3f8-8d7c-4167-9edc-0618abac7e96-kserve-provision-location\") pod \"36fbf3f8-8d7c-4167-9edc-0618abac7e96\" (UID: \"36fbf3f8-8d7c-4167-9edc-0618abac7e96\") "
Jul 31 15:28:34.409880 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.409858    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36fbf3f8-8d7c-4167-9edc-0618abac7e96-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "36fbf3f8-8d7c-4167-9edc-0618abac7e96" (UID: "36fbf3f8-8d7c-4167-9edc-0618abac7e96"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:28:34.409924 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.409868    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36fbf3f8-8d7c-4167-9edc-0618abac7e96-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "36fbf3f8-8d7c-4167-9edc-0618abac7e96" (UID: "36fbf3f8-8d7c-4167-9edc-0618abac7e96"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:28:34.410004 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.409989    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/36fbf3f8-8d7c-4167-9edc-0618abac7e96-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:28:34.410067 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.410009    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36fbf3f8-8d7c-4167-9edc-0618abac7e96-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:28:34.410067 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.410035    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36fbf3f8-8d7c-4167-9edc-0618abac7e96-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "36fbf3f8-8d7c-4167-9edc-0618abac7e96" (UID: "36fbf3f8-8d7c-4167-9edc-0618abac7e96"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:28:34.411874 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.411844    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36fbf3f8-8d7c-4167-9edc-0618abac7e96-kube-api-access-wl655" (OuterVolumeSpecName: "kube-api-access-wl655") pod "36fbf3f8-8d7c-4167-9edc-0618abac7e96" (UID: "36fbf3f8-8d7c-4167-9edc-0618abac7e96"). InnerVolumeSpecName "kube-api-access-wl655". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:28:34.411874 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.411856    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36fbf3f8-8d7c-4167-9edc-0618abac7e96-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "36fbf3f8-8d7c-4167-9edc-0618abac7e96" (UID: "36fbf3f8-8d7c-4167-9edc-0618abac7e96"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:28:34.510376 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.510296    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl655\" (UniqueName: \"kubernetes.io/projected/36fbf3f8-8d7c-4167-9edc-0618abac7e96-kube-api-access-wl655\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:28:34.510376 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.510325    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36fbf3f8-8d7c-4167-9edc-0618abac7e96-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:28:34.510376 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.510334    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36fbf3f8-8d7c-4167-9edc-0618abac7e96-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:28:34.844355 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.844262    2559 generic.go:356] "Generic (PLEG): container finished" podID="36fbf3f8-8d7c-4167-9edc-0618abac7e96" containerID="46dc09c5e3e534f59b38c638f9fc7efcbae4586bc9334f0d3022b7c3efb5bd74" exitCode=0
Jul 31 15:28:34.844355 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.844330    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"
Jul 31 15:28:34.844355 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.844342    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg" event={"ID":"36fbf3f8-8d7c-4167-9edc-0618abac7e96","Type":"ContainerDied","Data":"46dc09c5e3e534f59b38c638f9fc7efcbae4586bc9334f0d3022b7c3efb5bd74"}
Jul 31 15:28:34.844605 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.844406    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg" event={"ID":"36fbf3f8-8d7c-4167-9edc-0618abac7e96","Type":"ContainerDied","Data":"fc88d87bb692eaf6bc4c59bee804a4ab2c9f892b030ba8ff134ede863a49f80f"}
Jul 31 15:28:34.844605 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.844433    2559 scope.go:117] "RemoveContainer" containerID="f4ee7dd2b3b7396146e3786e9d91e18be67fffa292f948c72a065373ef93967b"
Jul 31 15:28:34.852858 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.852842    2559 scope.go:117] "RemoveContainer" containerID="46dc09c5e3e534f59b38c638f9fc7efcbae4586bc9334f0d3022b7c3efb5bd74"
Jul 31 15:28:34.860019 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.860002    2559 scope.go:117] "RemoveContainer" containerID="0e9ed0280e713783d16e54ee75552acb2072b00796db7ad3c146d1611b023679"
Jul 31 15:28:34.865935 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.865912    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"]
Jul 31 15:28:34.868614 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.868590    2559 scope.go:117] "RemoveContainer" containerID="f4ee7dd2b3b7396146e3786e9d91e18be67fffa292f948c72a065373ef93967b"
Jul 31 15:28:34.869176 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:28:34.869084    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4ee7dd2b3b7396146e3786e9d91e18be67fffa292f948c72a065373ef93967b\": container with ID starting with f4ee7dd2b3b7396146e3786e9d91e18be67fffa292f948c72a065373ef93967b not found: ID does not exist" containerID="f4ee7dd2b3b7396146e3786e9d91e18be67fffa292f948c72a065373ef93967b"
Jul 31 15:28:34.869176 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.869118    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4ee7dd2b3b7396146e3786e9d91e18be67fffa292f948c72a065373ef93967b"} err="failed to get container status \"f4ee7dd2b3b7396146e3786e9d91e18be67fffa292f948c72a065373ef93967b\": rpc error: code = NotFound desc = could not find container \"f4ee7dd2b3b7396146e3786e9d91e18be67fffa292f948c72a065373ef93967b\": container with ID starting with f4ee7dd2b3b7396146e3786e9d91e18be67fffa292f948c72a065373ef93967b not found: ID does not exist"
Jul 31 15:28:34.869176 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.869141    2559 scope.go:117] "RemoveContainer" containerID="46dc09c5e3e534f59b38c638f9fc7efcbae4586bc9334f0d3022b7c3efb5bd74"
Jul 31 15:28:34.869462 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:28:34.869443    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46dc09c5e3e534f59b38c638f9fc7efcbae4586bc9334f0d3022b7c3efb5bd74\": container with ID starting with 46dc09c5e3e534f59b38c638f9fc7efcbae4586bc9334f0d3022b7c3efb5bd74 not found: ID does not exist" containerID="46dc09c5e3e534f59b38c638f9fc7efcbae4586bc9334f0d3022b7c3efb5bd74"
Jul 31 15:28:34.869510 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.869469    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46dc09c5e3e534f59b38c638f9fc7efcbae4586bc9334f0d3022b7c3efb5bd74"} err="failed to get container status \"46dc09c5e3e534f59b38c638f9fc7efcbae4586bc9334f0d3022b7c3efb5bd74\": rpc error: code = NotFound desc = could not find container \"46dc09c5e3e534f59b38c638f9fc7efcbae4586bc9334f0d3022b7c3efb5bd74\": container with ID starting with 46dc09c5e3e534f59b38c638f9fc7efcbae4586bc9334f0d3022b7c3efb5bd74 not found: ID does not exist"
Jul 31 15:28:34.869510 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.869487    2559 scope.go:117] "RemoveContainer" containerID="0e9ed0280e713783d16e54ee75552acb2072b00796db7ad3c146d1611b023679"
Jul 31 15:28:34.869788 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:28:34.869767    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e9ed0280e713783d16e54ee75552acb2072b00796db7ad3c146d1611b023679\": container with ID starting with 0e9ed0280e713783d16e54ee75552acb2072b00796db7ad3c146d1611b023679 not found: ID does not exist" containerID="0e9ed0280e713783d16e54ee75552acb2072b00796db7ad3c146d1611b023679"
Jul 31 15:28:34.869849 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.869796    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e9ed0280e713783d16e54ee75552acb2072b00796db7ad3c146d1611b023679"} err="failed to get container status \"0e9ed0280e713783d16e54ee75552acb2072b00796db7ad3c146d1611b023679\": rpc error: code = NotFound desc = could not find container \"0e9ed0280e713783d16e54ee75552acb2072b00796db7ad3c146d1611b023679\": container with ID starting with 0e9ed0280e713783d16e54ee75552acb2072b00796db7ad3c146d1611b023679 not found: ID does not exist"
Jul 31 15:28:34.870367 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:34.870350    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zdpxg"]
Jul 31 15:28:35.753441 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.753403    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"]
Jul 31 15:28:35.753863 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.753662    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36fbf3f8-8d7c-4167-9edc-0618abac7e96" containerName="storage-initializer"
Jul 31 15:28:35.753863 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.753672    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="36fbf3f8-8d7c-4167-9edc-0618abac7e96" containerName="storage-initializer"
Jul 31 15:28:35.753863 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.753684    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c668a15-5b7c-4cec-8d02-403be047c243" containerName="storage-initializer"
Jul 31 15:28:35.753863 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.753689    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c668a15-5b7c-4cec-8d02-403be047c243" containerName="storage-initializer"
Jul 31 15:28:35.753863 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.753695    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36fbf3f8-8d7c-4167-9edc-0618abac7e96" containerName="kube-rbac-proxy"
Jul 31 15:28:35.753863 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.753701    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="36fbf3f8-8d7c-4167-9edc-0618abac7e96" containerName="kube-rbac-proxy"
Jul 31 15:28:35.753863 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.753710    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36fbf3f8-8d7c-4167-9edc-0618abac7e96" containerName="kserve-container"
Jul 31 15:28:35.753863 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.753715    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="36fbf3f8-8d7c-4167-9edc-0618abac7e96" containerName="kserve-container"
Jul 31 15:28:35.753863 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.753722    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c668a15-5b7c-4cec-8d02-403be047c243" containerName="kserve-container"
Jul 31 15:28:35.753863 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.753740    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c668a15-5b7c-4cec-8d02-403be047c243" containerName="kserve-container"
Jul 31 15:28:35.753863 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.753750    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c668a15-5b7c-4cec-8d02-403be047c243" containerName="kube-rbac-proxy"
Jul 31 15:28:35.753863 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.753756    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c668a15-5b7c-4cec-8d02-403be047c243" containerName="kube-rbac-proxy"
Jul 31 15:28:35.753863 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.753802    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c668a15-5b7c-4cec-8d02-403be047c243" containerName="kube-rbac-proxy"
Jul 31 15:28:35.753863 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.753810    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="36fbf3f8-8d7c-4167-9edc-0618abac7e96" containerName="kube-rbac-proxy"
Jul 31 15:28:35.753863 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.753815    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="36fbf3f8-8d7c-4167-9edc-0618abac7e96" containerName="kserve-container"
Jul 31 15:28:35.753863 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.753821    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c668a15-5b7c-4cec-8d02-403be047c243" containerName="kserve-container"
Jul 31 15:28:35.756684 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.756664    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"
Jul 31 15:28:35.759064 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.759035    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 15:28:35.759200 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.759077    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 31 15:28:35.759200 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.759098    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 15:28:35.759200 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.759136    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Jul 31 15:28:35.759577 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.759422    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:28:35.760250 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.760232    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dd67w\""
Jul 31 15:28:35.760395 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.760258    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 15:28:35.766534 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.766508    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"]
Jul 31 15:28:35.819927 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.819896    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86277011-5011-4eb8-b7a3-96ff8447f4da-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh\" (UID: \"86277011-5011-4eb8-b7a3-96ff8447f4da\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"
Jul 31 15:28:35.820118 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.819937    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxwnt\" (UniqueName: \"kubernetes.io/projected/86277011-5011-4eb8-b7a3-96ff8447f4da-kube-api-access-qxwnt\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh\" (UID: \"86277011-5011-4eb8-b7a3-96ff8447f4da\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"
Jul 31 15:28:35.820118 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.819980    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86277011-5011-4eb8-b7a3-96ff8447f4da-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh\" (UID: \"86277011-5011-4eb8-b7a3-96ff8447f4da\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"
Jul 31 15:28:35.820118 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.820032    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86277011-5011-4eb8-b7a3-96ff8447f4da-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh\" (UID: \"86277011-5011-4eb8-b7a3-96ff8447f4da\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"
Jul 31 15:28:35.820118 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.820063    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86277011-5011-4eb8-b7a3-96ff8447f4da-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh\" (UID: \"86277011-5011-4eb8-b7a3-96ff8447f4da\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"
Jul 31 15:28:35.921315 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.921274    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86277011-5011-4eb8-b7a3-96ff8447f4da-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh\" (UID: \"86277011-5011-4eb8-b7a3-96ff8447f4da\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"
Jul 31 15:28:35.921315 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.921315    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxwnt\" (UniqueName: \"kubernetes.io/projected/86277011-5011-4eb8-b7a3-96ff8447f4da-kube-api-access-qxwnt\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh\" (UID: \"86277011-5011-4eb8-b7a3-96ff8447f4da\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"
Jul 31 15:28:35.921690 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.921342    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86277011-5011-4eb8-b7a3-96ff8447f4da-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh\" (UID: \"86277011-5011-4eb8-b7a3-96ff8447f4da\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"
Jul 31 15:28:35.921690 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.921366    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86277011-5011-4eb8-b7a3-96ff8447f4da-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh\" (UID: \"86277011-5011-4eb8-b7a3-96ff8447f4da\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"
Jul 31 15:28:35.921690 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.921387    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86277011-5011-4eb8-b7a3-96ff8447f4da-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh\" (UID: \"86277011-5011-4eb8-b7a3-96ff8447f4da\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"
Jul 31 15:28:35.921858 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.921844    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86277011-5011-4eb8-b7a3-96ff8447f4da-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh\" (UID: \"86277011-5011-4eb8-b7a3-96ff8447f4da\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"
Jul 31 15:28:35.922239 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.922217    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86277011-5011-4eb8-b7a3-96ff8447f4da-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh\" (UID: \"86277011-5011-4eb8-b7a3-96ff8447f4da\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"
Jul 31 15:28:35.922278 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.922217    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86277011-5011-4eb8-b7a3-96ff8447f4da-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh\" (UID: \"86277011-5011-4eb8-b7a3-96ff8447f4da\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"
Jul 31 15:28:35.924039 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.924018    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86277011-5011-4eb8-b7a3-96ff8447f4da-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh\" (UID: \"86277011-5011-4eb8-b7a3-96ff8447f4da\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"
Jul 31 15:28:35.929705 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:35.929682    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxwnt\" (UniqueName: \"kubernetes.io/projected/86277011-5011-4eb8-b7a3-96ff8447f4da-kube-api-access-qxwnt\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh\" (UID: \"86277011-5011-4eb8-b7a3-96ff8447f4da\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"
Jul 31 15:28:36.069682 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:36.069591    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"
Jul 31 15:28:36.205646 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:36.205598    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"]
Jul 31 15:28:36.208402 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:28:36.208374    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86277011_5011_4eb8_b7a3_96ff8447f4da.slice/crio-eb7f3f1a8301f92a07f18b8e838fc4d368f68745ff18456e4a3a86cdada18279 WatchSource:0}: Error finding container eb7f3f1a8301f92a07f18b8e838fc4d368f68745ff18456e4a3a86cdada18279: Status 404 returned error can't find the container with id eb7f3f1a8301f92a07f18b8e838fc4d368f68745ff18456e4a3a86cdada18279
Jul 31 15:28:36.220445 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:36.220418    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36fbf3f8-8d7c-4167-9edc-0618abac7e96" path="/var/lib/kubelet/pods/36fbf3f8-8d7c-4167-9edc-0618abac7e96/volumes"
Jul 31 15:28:36.854179 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:36.854130    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh" event={"ID":"86277011-5011-4eb8-b7a3-96ff8447f4da","Type":"ContainerStarted","Data":"17788d2375432fb01936ad89317fe7ef93fc1a04cd4bf8ab856b4251e476b3e8"}
Jul 31 15:28:36.854603 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:36.854204    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh" event={"ID":"86277011-5011-4eb8-b7a3-96ff8447f4da","Type":"ContainerStarted","Data":"eb7f3f1a8301f92a07f18b8e838fc4d368f68745ff18456e4a3a86cdada18279"}
Jul 31 15:28:40.869861 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:40.869827    2559 generic.go:356] "Generic (PLEG): container finished" podID="86277011-5011-4eb8-b7a3-96ff8447f4da" containerID="17788d2375432fb01936ad89317fe7ef93fc1a04cd4bf8ab856b4251e476b3e8" exitCode=0
Jul 31 15:28:40.870249 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:40.869899    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh" event={"ID":"86277011-5011-4eb8-b7a3-96ff8447f4da","Type":"ContainerDied","Data":"17788d2375432fb01936ad89317fe7ef93fc1a04cd4bf8ab856b4251e476b3e8"}
Jul 31 15:28:41.876550 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:41.876509    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh" event={"ID":"86277011-5011-4eb8-b7a3-96ff8447f4da","Type":"ContainerStarted","Data":"33a2e88bd167ff70639c5a841fdf59b6d40145a3c89892d069afb8159c135cf2"}
Jul 31 15:28:41.877030 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:41.876582    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh" event={"ID":"86277011-5011-4eb8-b7a3-96ff8447f4da","Type":"ContainerStarted","Data":"73d4716353dc46684fca1429b4118856496d5a0a13ec6aca7b2725b55931572a"}
Jul 31 15:28:41.877030 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:41.876807    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"
Jul 31 15:28:41.877030 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:41.876926    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"
Jul 31 15:28:41.902606 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:41.902546    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh" podStartSLOduration=2.905855753 podStartE2EDuration="6.902525472s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.996669719s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:28:41.901490396 +0000 UTC m=+2418.213048043" watchObservedRunningTime="2026-07-31 15:28:41.902525472 +0000 UTC m=+2418.214083096"
Jul 31 15:28:47.886620 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:28:47.886535    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"
Jul 31 15:29:17.891055 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:17.891021    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"
Jul 31 15:29:25.834103 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:25.834068    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"]
Jul 31 15:29:25.835537 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:25.834300    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh" podUID="86277011-5011-4eb8-b7a3-96ff8447f4da" containerName="kserve-container" containerID="cri-o://73d4716353dc46684fca1429b4118856496d5a0a13ec6aca7b2725b55931572a" gracePeriod=30
Jul 31 15:29:25.835537 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:25.834382    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh" podUID="86277011-5011-4eb8-b7a3-96ff8447f4da" containerName="kube-rbac-proxy" containerID="cri-o://33a2e88bd167ff70639c5a841fdf59b6d40145a3c89892d069afb8159c135cf2" gracePeriod=30
Jul 31 15:29:26.036579 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:26.036550    2559 generic.go:356] "Generic (PLEG): container finished" podID="86277011-5011-4eb8-b7a3-96ff8447f4da" containerID="33a2e88bd167ff70639c5a841fdf59b6d40145a3c89892d069afb8159c135cf2" exitCode=2
Jul 31 15:29:26.036770 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:26.036627    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh" event={"ID":"86277011-5011-4eb8-b7a3-96ff8447f4da","Type":"ContainerDied","Data":"33a2e88bd167ff70639c5a841fdf59b6d40145a3c89892d069afb8159c135cf2"}
Jul 31 15:29:27.882243 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:27.882198    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh" podUID="86277011-5011-4eb8-b7a3-96ff8447f4da" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Jul 31 15:29:31.069684 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:31.069641    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"]
Jul 31 15:29:31.073186 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:31.073162    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"
Jul 31 15:29:31.076265 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:31.076241    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Jul 31 15:29:31.076686 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:31.076666    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 31 15:29:31.086859 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:31.086834    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"]
Jul 31 15:29:31.155205 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:31.155168    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckfhv\" (UniqueName: \"kubernetes.io/projected/a68ef276-e099-43e3-b088-fe194f57f0ca-kube-api-access-ckfhv\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv\" (UID: \"a68ef276-e099-43e3-b088-fe194f57f0ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"
Jul 31 15:29:31.155377 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:31.155220    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a68ef276-e099-43e3-b088-fe194f57f0ca-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv\" (UID: \"a68ef276-e099-43e3-b088-fe194f57f0ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"
Jul 31 15:29:31.155377 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:31.155242    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a68ef276-e099-43e3-b088-fe194f57f0ca-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv\" (UID: \"a68ef276-e099-43e3-b088-fe194f57f0ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"
Jul 31 15:29:31.155377 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:31.155291    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a68ef276-e099-43e3-b088-fe194f57f0ca-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv\" (UID: \"a68ef276-e099-43e3-b088-fe194f57f0ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"
Jul 31 15:29:31.155534 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:31.155369    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a68ef276-e099-43e3-b088-fe194f57f0ca-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv\" (UID: \"a68ef276-e099-43e3-b088-fe194f57f0ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"
Jul 31 15:29:31.256060 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:31.256029    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckfhv\" (UniqueName: \"kubernetes.io/projected/a68ef276-e099-43e3-b088-fe194f57f0ca-kube-api-access-ckfhv\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv\" (UID: \"a68ef276-e099-43e3-b088-fe194f57f0ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"
Jul 31 15:29:31.256275 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:31.256106    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a68ef276-e099-43e3-b088-fe194f57f0ca-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv\" (UID: \"a68ef276-e099-43e3-b088-fe194f57f0ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"
Jul 31 15:29:31.256275 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:31.256136    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a68ef276-e099-43e3-b088-fe194f57f0ca-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv\" (UID: \"a68ef276-e099-43e3-b088-fe194f57f0ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"
Jul 31 15:29:31.256275 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:31.256161    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a68ef276-e099-43e3-b088-fe194f57f0ca-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv\" (UID: \"a68ef276-e099-43e3-b088-fe194f57f0ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"
Jul 31 15:29:31.256275 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:31.256209    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a68ef276-e099-43e3-b088-fe194f57f0ca-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv\" (UID: \"a68ef276-e099-43e3-b088-fe194f57f0ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"
Jul 31 15:29:31.256782 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:31.256756    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a68ef276-e099-43e3-b088-fe194f57f0ca-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv\" (UID: \"a68ef276-e099-43e3-b088-fe194f57f0ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"
Jul 31 15:29:31.257185 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:31.257159    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a68ef276-e099-43e3-b088-fe194f57f0ca-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv\" (UID: \"a68ef276-e099-43e3-b088-fe194f57f0ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"
Jul 31 15:29:31.257271 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:31.257184    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a68ef276-e099-43e3-b088-fe194f57f0ca-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv\" (UID: \"a68ef276-e099-43e3-b088-fe194f57f0ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"
Jul 31 15:29:31.259175 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:31.259157    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a68ef276-e099-43e3-b088-fe194f57f0ca-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv\" (UID: \"a68ef276-e099-43e3-b088-fe194f57f0ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"
Jul 31 15:29:31.265871 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:31.265846    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckfhv\" (UniqueName: \"kubernetes.io/projected/a68ef276-e099-43e3-b088-fe194f57f0ca-kube-api-access-ckfhv\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv\" (UID: \"a68ef276-e099-43e3-b088-fe194f57f0ca\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"
Jul 31 15:29:31.383233 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:31.383188    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"
Jul 31 15:29:31.521874 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:31.521829    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"]
Jul 31 15:29:31.526791 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:29:31.526762    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda68ef276_e099_43e3_b088_fe194f57f0ca.slice/crio-5d315fca4fc3d93bd81f6c40e673a6950b5b95b53677c1e4e4f398c006aba8d1 WatchSource:0}: Error finding container 5d315fca4fc3d93bd81f6c40e673a6950b5b95b53677c1e4e4f398c006aba8d1: Status 404 returned error can't find the container with id 5d315fca4fc3d93bd81f6c40e673a6950b5b95b53677c1e4e4f398c006aba8d1
Jul 31 15:29:32.064199 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:32.064154    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv" event={"ID":"a68ef276-e099-43e3-b088-fe194f57f0ca","Type":"ContainerStarted","Data":"595545e4e1eff23da6dd38f332fbd96eaebf018f91c63626dfd594d6958a7074"}
Jul 31 15:29:32.064418 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:32.064219    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv" event={"ID":"a68ef276-e099-43e3-b088-fe194f57f0ca","Type":"ContainerStarted","Data":"5d315fca4fc3d93bd81f6c40e673a6950b5b95b53677c1e4e4f398c006aba8d1"}
Jul 31 15:29:32.610388 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:32.610363    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"
Jul 31 15:29:32.670162 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:32.670063    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86277011-5011-4eb8-b7a3-96ff8447f4da-kube-rbac-proxy-sar-config\") pod \"86277011-5011-4eb8-b7a3-96ff8447f4da\" (UID: \"86277011-5011-4eb8-b7a3-96ff8447f4da\") "
Jul 31 15:29:32.670162 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:32.670146    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86277011-5011-4eb8-b7a3-96ff8447f4da-proxy-tls\") pod \"86277011-5011-4eb8-b7a3-96ff8447f4da\" (UID: \"86277011-5011-4eb8-b7a3-96ff8447f4da\") "
Jul 31 15:29:32.670373 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:32.670171    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86277011-5011-4eb8-b7a3-96ff8447f4da-cabundle-cert\") pod \"86277011-5011-4eb8-b7a3-96ff8447f4da\" (UID: \"86277011-5011-4eb8-b7a3-96ff8447f4da\") "
Jul 31 15:29:32.670373 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:32.670211    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86277011-5011-4eb8-b7a3-96ff8447f4da-kserve-provision-location\") pod \"86277011-5011-4eb8-b7a3-96ff8447f4da\" (UID: \"86277011-5011-4eb8-b7a3-96ff8447f4da\") "
Jul 31 15:29:32.670373 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:32.670258    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxwnt\" (UniqueName: \"kubernetes.io/projected/86277011-5011-4eb8-b7a3-96ff8447f4da-kube-api-access-qxwnt\") pod \"86277011-5011-4eb8-b7a3-96ff8447f4da\" (UID: \"86277011-5011-4eb8-b7a3-96ff8447f4da\") "
Jul 31 15:29:32.670560 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:32.670530    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86277011-5011-4eb8-b7a3-96ff8447f4da-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "86277011-5011-4eb8-b7a3-96ff8447f4da" (UID: "86277011-5011-4eb8-b7a3-96ff8447f4da"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:29:32.670607 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:32.670536    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86277011-5011-4eb8-b7a3-96ff8447f4da-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "86277011-5011-4eb8-b7a3-96ff8447f4da" (UID: "86277011-5011-4eb8-b7a3-96ff8447f4da"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:29:32.670652 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:32.670600    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86277011-5011-4eb8-b7a3-96ff8447f4da-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "86277011-5011-4eb8-b7a3-96ff8447f4da" (UID: "86277011-5011-4eb8-b7a3-96ff8447f4da"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:29:32.672391 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:32.672372    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86277011-5011-4eb8-b7a3-96ff8447f4da-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "86277011-5011-4eb8-b7a3-96ff8447f4da" (UID: "86277011-5011-4eb8-b7a3-96ff8447f4da"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:29:32.672596 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:32.672578    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86277011-5011-4eb8-b7a3-96ff8447f4da-kube-api-access-qxwnt" (OuterVolumeSpecName: "kube-api-access-qxwnt") pod "86277011-5011-4eb8-b7a3-96ff8447f4da" (UID: "86277011-5011-4eb8-b7a3-96ff8447f4da"). InnerVolumeSpecName "kube-api-access-qxwnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:29:32.771144 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:32.771105    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86277011-5011-4eb8-b7a3-96ff8447f4da-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:29:32.771144 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:32.771139    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxwnt\" (UniqueName: \"kubernetes.io/projected/86277011-5011-4eb8-b7a3-96ff8447f4da-kube-api-access-qxwnt\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:29:32.771144 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:32.771149    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86277011-5011-4eb8-b7a3-96ff8447f4da-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:29:32.771381 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:32.771160    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86277011-5011-4eb8-b7a3-96ff8447f4da-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:29:32.771381 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:32.771170    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86277011-5011-4eb8-b7a3-96ff8447f4da-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:29:33.070165 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:33.070071    2559 generic.go:356] "Generic (PLEG): container finished" podID="86277011-5011-4eb8-b7a3-96ff8447f4da" containerID="73d4716353dc46684fca1429b4118856496d5a0a13ec6aca7b2725b55931572a" exitCode=0
Jul 31 15:29:33.070165 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:33.070133    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"
Jul 31 15:29:33.070351 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:33.070156    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh" event={"ID":"86277011-5011-4eb8-b7a3-96ff8447f4da","Type":"ContainerDied","Data":"73d4716353dc46684fca1429b4118856496d5a0a13ec6aca7b2725b55931572a"}
Jul 31 15:29:33.070351 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:33.070221    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh" event={"ID":"86277011-5011-4eb8-b7a3-96ff8447f4da","Type":"ContainerDied","Data":"eb7f3f1a8301f92a07f18b8e838fc4d368f68745ff18456e4a3a86cdada18279"}
Jul 31 15:29:33.070351 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:33.070262    2559 scope.go:117] "RemoveContainer" containerID="33a2e88bd167ff70639c5a841fdf59b6d40145a3c89892d069afb8159c135cf2"
Jul 31 15:29:33.079140 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:33.079118    2559 scope.go:117] "RemoveContainer" containerID="73d4716353dc46684fca1429b4118856496d5a0a13ec6aca7b2725b55931572a"
Jul 31 15:29:33.086923 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:33.086902    2559 scope.go:117] "RemoveContainer" containerID="17788d2375432fb01936ad89317fe7ef93fc1a04cd4bf8ab856b4251e476b3e8"
Jul 31 15:29:33.093263 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:33.093241    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"]
Jul 31 15:29:33.095545 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:33.095480    2559 scope.go:117] "RemoveContainer" containerID="33a2e88bd167ff70639c5a841fdf59b6d40145a3c89892d069afb8159c135cf2"
Jul 31 15:29:33.095828 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:29:33.095806    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33a2e88bd167ff70639c5a841fdf59b6d40145a3c89892d069afb8159c135cf2\": container with ID starting with 33a2e88bd167ff70639c5a841fdf59b6d40145a3c89892d069afb8159c135cf2 not found: ID does not exist" containerID="33a2e88bd167ff70639c5a841fdf59b6d40145a3c89892d069afb8159c135cf2"
Jul 31 15:29:33.095915 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:33.095843    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a2e88bd167ff70639c5a841fdf59b6d40145a3c89892d069afb8159c135cf2"} err="failed to get container status \"33a2e88bd167ff70639c5a841fdf59b6d40145a3c89892d069afb8159c135cf2\": rpc error: code = NotFound desc = could not find container \"33a2e88bd167ff70639c5a841fdf59b6d40145a3c89892d069afb8159c135cf2\": container with ID starting with 33a2e88bd167ff70639c5a841fdf59b6d40145a3c89892d069afb8159c135cf2 not found: ID does not exist"
Jul 31 15:29:33.095915 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:33.095861    2559 scope.go:117] "RemoveContainer" containerID="73d4716353dc46684fca1429b4118856496d5a0a13ec6aca7b2725b55931572a"
Jul 31 15:29:33.096084 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:33.096063    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m94gh"]
Jul 31 15:29:33.096171 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:29:33.096155    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d4716353dc46684fca1429b4118856496d5a0a13ec6aca7b2725b55931572a\": container with ID starting with 73d4716353dc46684fca1429b4118856496d5a0a13ec6aca7b2725b55931572a not found: ID does not exist" containerID="73d4716353dc46684fca1429b4118856496d5a0a13ec6aca7b2725b55931572a"
Jul 31 15:29:33.096208 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:33.096178    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d4716353dc46684fca1429b4118856496d5a0a13ec6aca7b2725b55931572a"} err="failed to get container status \"73d4716353dc46684fca1429b4118856496d5a0a13ec6aca7b2725b55931572a\": rpc error: code = NotFound desc = could not find container \"73d4716353dc46684fca1429b4118856496d5a0a13ec6aca7b2725b55931572a\": container with ID starting with 73d4716353dc46684fca1429b4118856496d5a0a13ec6aca7b2725b55931572a not found: ID does not exist"
Jul 31 15:29:33.096208 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:33.096195    2559 scope.go:117] "RemoveContainer" containerID="17788d2375432fb01936ad89317fe7ef93fc1a04cd4bf8ab856b4251e476b3e8"
Jul 31 15:29:33.096465 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:29:33.096448    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17788d2375432fb01936ad89317fe7ef93fc1a04cd4bf8ab856b4251e476b3e8\": container with ID starting with 17788d2375432fb01936ad89317fe7ef93fc1a04cd4bf8ab856b4251e476b3e8 not found: ID does not exist" containerID="17788d2375432fb01936ad89317fe7ef93fc1a04cd4bf8ab856b4251e476b3e8"
Jul 31 15:29:33.096506 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:33.096470    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17788d2375432fb01936ad89317fe7ef93fc1a04cd4bf8ab856b4251e476b3e8"} err="failed to get container status \"17788d2375432fb01936ad89317fe7ef93fc1a04cd4bf8ab856b4251e476b3e8\": rpc error: code = NotFound desc = could not find container \"17788d2375432fb01936ad89317fe7ef93fc1a04cd4bf8ab856b4251e476b3e8\": container with ID starting with 17788d2375432fb01936ad89317fe7ef93fc1a04cd4bf8ab856b4251e476b3e8 not found: ID does not exist"
Jul 31 15:29:34.223975 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:34.222217    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86277011-5011-4eb8-b7a3-96ff8447f4da" path="/var/lib/kubelet/pods/86277011-5011-4eb8-b7a3-96ff8447f4da/volumes"
Jul 31 15:29:36.082684 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:36.082650    2559 generic.go:356] "Generic (PLEG): container finished" podID="a68ef276-e099-43e3-b088-fe194f57f0ca" containerID="595545e4e1eff23da6dd38f332fbd96eaebf018f91c63626dfd594d6958a7074" exitCode=0
Jul 31 15:29:36.083070 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:36.082724    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv" event={"ID":"a68ef276-e099-43e3-b088-fe194f57f0ca","Type":"ContainerDied","Data":"595545e4e1eff23da6dd38f332fbd96eaebf018f91c63626dfd594d6958a7074"}
Jul 31 15:29:37.088887 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:37.088853    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv" event={"ID":"a68ef276-e099-43e3-b088-fe194f57f0ca","Type":"ContainerStarted","Data":"71ba5438a02eccb42ffabd8ec44f59401748aad34a9e1f47f1d8073c8c2b7657"}
Jul 31 15:29:37.089294 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:37.088920    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv" event={"ID":"a68ef276-e099-43e3-b088-fe194f57f0ca","Type":"ContainerStarted","Data":"39b0f69d571f6cbe4507b5c0035893509be4b071985c2bcba5402d2cc5b3c1ae"}
Jul 31 15:29:37.089294 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:37.089179    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"
Jul 31 15:29:37.111192 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:37.111145    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv" podStartSLOduration=2.178323606 podStartE2EDuration="6.11112931s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.932805704s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:29:37.110011906 +0000 UTC m=+2473.421569556" watchObservedRunningTime="2026-07-31 15:29:37.11112931 +0000 UTC m=+2473.422686932"
Jul 31 15:29:38.092410 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:38.092380    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"
Jul 31 15:29:44.102918 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:29:44.102879    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"
Jul 31 15:30:14.106833 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:14.106794    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"
Jul 31 15:30:21.105067 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:21.105031    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"]
Jul 31 15:30:21.105501 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:21.105273    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv" podUID="a68ef276-e099-43e3-b088-fe194f57f0ca" containerName="kserve-container" containerID="cri-o://39b0f69d571f6cbe4507b5c0035893509be4b071985c2bcba5402d2cc5b3c1ae" gracePeriod=30
Jul 31 15:30:21.105501 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:21.105377    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv" podUID="a68ef276-e099-43e3-b088-fe194f57f0ca" containerName="kube-rbac-proxy" containerID="cri-o://71ba5438a02eccb42ffabd8ec44f59401748aad34a9e1f47f1d8073c8c2b7657" gracePeriod=30
Jul 31 15:30:21.255171 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:21.255132    2559 generic.go:356] "Generic (PLEG): container finished" podID="a68ef276-e099-43e3-b088-fe194f57f0ca" containerID="71ba5438a02eccb42ffabd8ec44f59401748aad34a9e1f47f1d8073c8c2b7657" exitCode=2
Jul 31 15:30:21.255333 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:21.255205    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv" event={"ID":"a68ef276-e099-43e3-b088-fe194f57f0ca","Type":"ContainerDied","Data":"71ba5438a02eccb42ffabd8ec44f59401748aad34a9e1f47f1d8073c8c2b7657"}
Jul 31 15:30:24.097309 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:24.097266    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv" podUID="a68ef276-e099-43e3-b088-fe194f57f0ca" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Jul 31 15:30:24.103569 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:24.103533    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv" podUID="a68ef276-e099-43e3-b088-fe194f57f0ca" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.50:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 31 15:30:26.281376 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.281340    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"]
Jul 31 15:30:26.281781 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.281594    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86277011-5011-4eb8-b7a3-96ff8447f4da" containerName="storage-initializer"
Jul 31 15:30:26.281781 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.281604    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="86277011-5011-4eb8-b7a3-96ff8447f4da" containerName="storage-initializer"
Jul 31 15:30:26.281781 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.281615    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86277011-5011-4eb8-b7a3-96ff8447f4da" containerName="kube-rbac-proxy"
Jul 31 15:30:26.281781 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.281621    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="86277011-5011-4eb8-b7a3-96ff8447f4da" containerName="kube-rbac-proxy"
Jul 31 15:30:26.281781 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.281629    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86277011-5011-4eb8-b7a3-96ff8447f4da" containerName="kserve-container"
Jul 31 15:30:26.281781 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.281634    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="86277011-5011-4eb8-b7a3-96ff8447f4da" containerName="kserve-container"
Jul 31 15:30:26.281781 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.281691    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="86277011-5011-4eb8-b7a3-96ff8447f4da" containerName="kserve-container"
Jul 31 15:30:26.281781 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.281698    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="86277011-5011-4eb8-b7a3-96ff8447f4da" containerName="kube-rbac-proxy"
Jul 31 15:30:26.284680 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.284660    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"
Jul 31 15:30:26.286898 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.286876    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Jul 31 15:30:26.286898 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.286888    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Jul 31 15:30:26.301868 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.301842    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"]
Jul 31 15:30:26.373308 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.373270    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsr65\" (UniqueName: \"kubernetes.io/projected/c01b0f83-8f47-4505-8887-181df5342228-kube-api-access-jsr65\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-dctqq\" (UID: \"c01b0f83-8f47-4505-8887-181df5342228\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"
Jul 31 15:30:26.373551 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.373320    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c01b0f83-8f47-4505-8887-181df5342228-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-dctqq\" (UID: \"c01b0f83-8f47-4505-8887-181df5342228\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"
Jul 31 15:30:26.373551 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.373373    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c01b0f83-8f47-4505-8887-181df5342228-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-dctqq\" (UID: \"c01b0f83-8f47-4505-8887-181df5342228\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"
Jul 31 15:30:26.373551 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.373408    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c01b0f83-8f47-4505-8887-181df5342228-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-dctqq\" (UID: \"c01b0f83-8f47-4505-8887-181df5342228\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"
Jul 31 15:30:26.373551 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.373478    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c01b0f83-8f47-4505-8887-181df5342228-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-dctqq\" (UID: \"c01b0f83-8f47-4505-8887-181df5342228\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"
Jul 31 15:30:26.474596 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.474557    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsr65\" (UniqueName: \"kubernetes.io/projected/c01b0f83-8f47-4505-8887-181df5342228-kube-api-access-jsr65\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-dctqq\" (UID: \"c01b0f83-8f47-4505-8887-181df5342228\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"
Jul 31 15:30:26.474596 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.474595    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c01b0f83-8f47-4505-8887-181df5342228-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-dctqq\" (UID: \"c01b0f83-8f47-4505-8887-181df5342228\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"
Jul 31 15:30:26.474889 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.474629    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c01b0f83-8f47-4505-8887-181df5342228-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-dctqq\" (UID: \"c01b0f83-8f47-4505-8887-181df5342228\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"
Jul 31 15:30:26.474889 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.474663    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c01b0f83-8f47-4505-8887-181df5342228-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-dctqq\" (UID: \"c01b0f83-8f47-4505-8887-181df5342228\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"
Jul 31 15:30:26.474889 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.474719    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c01b0f83-8f47-4505-8887-181df5342228-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-dctqq\" (UID: \"c01b0f83-8f47-4505-8887-181df5342228\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"
Jul 31 15:30:26.474889 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:30:26.474853    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-serving-cert: secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Jul 31 15:30:26.475097 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:30:26.474940    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c01b0f83-8f47-4505-8887-181df5342228-proxy-tls podName:c01b0f83-8f47-4505-8887-181df5342228 nodeName:}" failed. No retries permitted until 2026-07-31 15:30:26.974915761 +0000 UTC m=+2523.286473365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c01b0f83-8f47-4505-8887-181df5342228-proxy-tls") pod "isvc-xgboost-runtime-predictor-54dd488c85-dctqq" (UID: "c01b0f83-8f47-4505-8887-181df5342228") : secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Jul 31 15:30:26.475167 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.475125    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c01b0f83-8f47-4505-8887-181df5342228-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-dctqq\" (UID: \"c01b0f83-8f47-4505-8887-181df5342228\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"
Jul 31 15:30:26.475291 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.475271    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c01b0f83-8f47-4505-8887-181df5342228-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-dctqq\" (UID: \"c01b0f83-8f47-4505-8887-181df5342228\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"
Jul 31 15:30:26.475348 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.475333    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c01b0f83-8f47-4505-8887-181df5342228-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-dctqq\" (UID: \"c01b0f83-8f47-4505-8887-181df5342228\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"
Jul 31 15:30:26.486075 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.486059    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsr65\" (UniqueName: \"kubernetes.io/projected/c01b0f83-8f47-4505-8887-181df5342228-kube-api-access-jsr65\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-dctqq\" (UID: \"c01b0f83-8f47-4505-8887-181df5342228\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"
Jul 31 15:30:26.978686 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.978647    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c01b0f83-8f47-4505-8887-181df5342228-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-dctqq\" (UID: \"c01b0f83-8f47-4505-8887-181df5342228\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"
Jul 31 15:30:26.981246 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:26.981225    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c01b0f83-8f47-4505-8887-181df5342228-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-dctqq\" (UID: \"c01b0f83-8f47-4505-8887-181df5342228\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"
Jul 31 15:30:27.194890 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:27.194842    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"
Jul 31 15:30:27.328342 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:27.328151    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"]
Jul 31 15:30:27.331204 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:30:27.331174    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc01b0f83_8f47_4505_8887_181df5342228.slice/crio-b632076860beac119e120b25d85f4605e017109d46575860f77823c87559a1b4 WatchSource:0}: Error finding container b632076860beac119e120b25d85f4605e017109d46575860f77823c87559a1b4: Status 404 returned error can't find the container with id b632076860beac119e120b25d85f4605e017109d46575860f77823c87559a1b4
Jul 31 15:30:27.946529 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:27.946496    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"
Jul 31 15:30:27.987694 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:27.987665    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a68ef276-e099-43e3-b088-fe194f57f0ca-proxy-tls\") pod \"a68ef276-e099-43e3-b088-fe194f57f0ca\" (UID: \"a68ef276-e099-43e3-b088-fe194f57f0ca\") "
Jul 31 15:30:27.987923 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:27.987778    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a68ef276-e099-43e3-b088-fe194f57f0ca-kserve-provision-location\") pod \"a68ef276-e099-43e3-b088-fe194f57f0ca\" (UID: \"a68ef276-e099-43e3-b088-fe194f57f0ca\") "
Jul 31 15:30:27.987923 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:27.987807    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a68ef276-e099-43e3-b088-fe194f57f0ca-cabundle-cert\") pod \"a68ef276-e099-43e3-b088-fe194f57f0ca\" (UID: \"a68ef276-e099-43e3-b088-fe194f57f0ca\") "
Jul 31 15:30:27.987923 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:27.987836    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a68ef276-e099-43e3-b088-fe194f57f0ca-kube-rbac-proxy-sar-config\") pod \"a68ef276-e099-43e3-b088-fe194f57f0ca\" (UID: \"a68ef276-e099-43e3-b088-fe194f57f0ca\") "
Jul 31 15:30:27.987923 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:27.987861    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckfhv\" (UniqueName: \"kubernetes.io/projected/a68ef276-e099-43e3-b088-fe194f57f0ca-kube-api-access-ckfhv\") pod \"a68ef276-e099-43e3-b088-fe194f57f0ca\" (UID: \"a68ef276-e099-43e3-b088-fe194f57f0ca\") "
Jul 31 15:30:27.988137 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:27.988108    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a68ef276-e099-43e3-b088-fe194f57f0ca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a68ef276-e099-43e3-b088-fe194f57f0ca" (UID: "a68ef276-e099-43e3-b088-fe194f57f0ca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:30:27.988137 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:27.988125    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a68ef276-e099-43e3-b088-fe194f57f0ca-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a68ef276-e099-43e3-b088-fe194f57f0ca" (UID: "a68ef276-e099-43e3-b088-fe194f57f0ca"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:30:27.988258 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:27.988181    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a68ef276-e099-43e3-b088-fe194f57f0ca-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a68ef276-e099-43e3-b088-fe194f57f0ca" (UID: "a68ef276-e099-43e3-b088-fe194f57f0ca"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:30:27.990244 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:27.990213    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a68ef276-e099-43e3-b088-fe194f57f0ca-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a68ef276-e099-43e3-b088-fe194f57f0ca" (UID: "a68ef276-e099-43e3-b088-fe194f57f0ca"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:30:27.990244 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:27.990227    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a68ef276-e099-43e3-b088-fe194f57f0ca-kube-api-access-ckfhv" (OuterVolumeSpecName: "kube-api-access-ckfhv") pod "a68ef276-e099-43e3-b088-fe194f57f0ca" (UID: "a68ef276-e099-43e3-b088-fe194f57f0ca"). InnerVolumeSpecName "kube-api-access-ckfhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:30:28.089246 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:28.089161    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a68ef276-e099-43e3-b088-fe194f57f0ca-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:30:28.089246 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:28.089190    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a68ef276-e099-43e3-b088-fe194f57f0ca-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:30:28.089246 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:28.089199    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a68ef276-e099-43e3-b088-fe194f57f0ca-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:30:28.089246 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:28.089207    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a68ef276-e099-43e3-b088-fe194f57f0ca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:30:28.089246 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:28.089215    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckfhv\" (UniqueName: \"kubernetes.io/projected/a68ef276-e099-43e3-b088-fe194f57f0ca-kube-api-access-ckfhv\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:30:28.282198 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:28.282163    2559 generic.go:356] "Generic (PLEG): container finished" podID="a68ef276-e099-43e3-b088-fe194f57f0ca" containerID="39b0f69d571f6cbe4507b5c0035893509be4b071985c2bcba5402d2cc5b3c1ae" exitCode=0
Jul 31 15:30:28.282383 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:28.282219    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"
Jul 31 15:30:28.282383 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:28.282250    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv" event={"ID":"a68ef276-e099-43e3-b088-fe194f57f0ca","Type":"ContainerDied","Data":"39b0f69d571f6cbe4507b5c0035893509be4b071985c2bcba5402d2cc5b3c1ae"}
Jul 31 15:30:28.282383 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:28.282310    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv" event={"ID":"a68ef276-e099-43e3-b088-fe194f57f0ca","Type":"ContainerDied","Data":"5d315fca4fc3d93bd81f6c40e673a6950b5b95b53677c1e4e4f398c006aba8d1"}
Jul 31 15:30:28.282383 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:28.282349    2559 scope.go:117] "RemoveContainer" containerID="71ba5438a02eccb42ffabd8ec44f59401748aad34a9e1f47f1d8073c8c2b7657"
Jul 31 15:30:28.284160 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:28.284129    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq" event={"ID":"c01b0f83-8f47-4505-8887-181df5342228","Type":"ContainerStarted","Data":"ca83d7af48f1c4155a2a575d128a89ede52148a17e0c063b0a9e7efd4bbab0ec"}
Jul 31 15:30:28.284308 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:28.284195    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq" event={"ID":"c01b0f83-8f47-4505-8887-181df5342228","Type":"ContainerStarted","Data":"b632076860beac119e120b25d85f4605e017109d46575860f77823c87559a1b4"}
Jul 31 15:30:28.290589 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:28.290572    2559 scope.go:117] "RemoveContainer" containerID="39b0f69d571f6cbe4507b5c0035893509be4b071985c2bcba5402d2cc5b3c1ae"
Jul 31 15:30:28.297828 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:28.297809    2559 scope.go:117] "RemoveContainer" containerID="595545e4e1eff23da6dd38f332fbd96eaebf018f91c63626dfd594d6958a7074"
Jul 31 15:30:28.305514 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:28.305418    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"]
Jul 31 15:30:28.305568 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:28.305554    2559 scope.go:117] "RemoveContainer" containerID="71ba5438a02eccb42ffabd8ec44f59401748aad34a9e1f47f1d8073c8c2b7657"
Jul 31 15:30:28.305858 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:30:28.305840    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71ba5438a02eccb42ffabd8ec44f59401748aad34a9e1f47f1d8073c8c2b7657\": container with ID starting with 71ba5438a02eccb42ffabd8ec44f59401748aad34a9e1f47f1d8073c8c2b7657 not found: ID does not exist" containerID="71ba5438a02eccb42ffabd8ec44f59401748aad34a9e1f47f1d8073c8c2b7657"
Jul 31 15:30:28.305910 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:28.305868    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ba5438a02eccb42ffabd8ec44f59401748aad34a9e1f47f1d8073c8c2b7657"} err="failed to get container status \"71ba5438a02eccb42ffabd8ec44f59401748aad34a9e1f47f1d8073c8c2b7657\": rpc error: code = NotFound desc = could not find container \"71ba5438a02eccb42ffabd8ec44f59401748aad34a9e1f47f1d8073c8c2b7657\": container with ID starting with 71ba5438a02eccb42ffabd8ec44f59401748aad34a9e1f47f1d8073c8c2b7657 not found: ID does not exist"
Jul 31 15:30:28.305910 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:28.305884    2559 scope.go:117] "RemoveContainer" containerID="39b0f69d571f6cbe4507b5c0035893509be4b071985c2bcba5402d2cc5b3c1ae"
Jul 31 15:30:28.306474 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:30:28.306271    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39b0f69d571f6cbe4507b5c0035893509be4b071985c2bcba5402d2cc5b3c1ae\": container with ID starting with 39b0f69d571f6cbe4507b5c0035893509be4b071985c2bcba5402d2cc5b3c1ae not found: ID does not exist" containerID="39b0f69d571f6cbe4507b5c0035893509be4b071985c2bcba5402d2cc5b3c1ae"
Jul 31 15:30:28.306474 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:28.306326    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39b0f69d571f6cbe4507b5c0035893509be4b071985c2bcba5402d2cc5b3c1ae"} err="failed to get container status \"39b0f69d571f6cbe4507b5c0035893509be4b071985c2bcba5402d2cc5b3c1ae\": rpc error: code = NotFound desc = could not find container \"39b0f69d571f6cbe4507b5c0035893509be4b071985c2bcba5402d2cc5b3c1ae\": container with ID starting with 39b0f69d571f6cbe4507b5c0035893509be4b071985c2bcba5402d2cc5b3c1ae not found: ID does not exist"
Jul 31 15:30:28.306474 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:28.306362    2559 scope.go:117] "RemoveContainer" containerID="595545e4e1eff23da6dd38f332fbd96eaebf018f91c63626dfd594d6958a7074"
Jul 31 15:30:28.306820 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:30:28.306799    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"595545e4e1eff23da6dd38f332fbd96eaebf018f91c63626dfd594d6958a7074\": container with ID starting with 595545e4e1eff23da6dd38f332fbd96eaebf018f91c63626dfd594d6958a7074 not found: ID does not exist" containerID="595545e4e1eff23da6dd38f332fbd96eaebf018f91c63626dfd594d6958a7074"
Jul 31 15:30:28.306896 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:28.306827    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"595545e4e1eff23da6dd38f332fbd96eaebf018f91c63626dfd594d6958a7074"} err="failed to get container status \"595545e4e1eff23da6dd38f332fbd96eaebf018f91c63626dfd594d6958a7074\": rpc error: code = NotFound desc = could not find container \"595545e4e1eff23da6dd38f332fbd96eaebf018f91c63626dfd594d6958a7074\": container with ID starting with 595545e4e1eff23da6dd38f332fbd96eaebf018f91c63626dfd594d6958a7074 not found: ID does not exist"
Jul 31 15:30:28.308007 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:28.307989    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dvvcv"]
Jul 31 15:30:30.221206 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:30.221168    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a68ef276-e099-43e3-b088-fe194f57f0ca" path="/var/lib/kubelet/pods/a68ef276-e099-43e3-b088-fe194f57f0ca/volumes"
Jul 31 15:30:32.300745 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:32.300697    2559 generic.go:356] "Generic (PLEG): container finished" podID="c01b0f83-8f47-4505-8887-181df5342228" containerID="ca83d7af48f1c4155a2a575d128a89ede52148a17e0c063b0a9e7efd4bbab0ec" exitCode=0
Jul 31 15:30:32.301147 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:32.300768    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq" event={"ID":"c01b0f83-8f47-4505-8887-181df5342228","Type":"ContainerDied","Data":"ca83d7af48f1c4155a2a575d128a89ede52148a17e0c063b0a9e7efd4bbab0ec"}
Jul 31 15:30:33.307069 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:33.307027    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq" event={"ID":"c01b0f83-8f47-4505-8887-181df5342228","Type":"ContainerStarted","Data":"3ba2220c4daee892cd352b6ef71bef65d86335464d6a0bdd214a04ec7d01f55b"}
Jul 31 15:30:33.307500 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:33.307092    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq" event={"ID":"c01b0f83-8f47-4505-8887-181df5342228","Type":"ContainerStarted","Data":"344569c3b8ba85b0396d6ec1ca1b3aad74c7f3a6cf9424e8e2fe90c7b87cd222"}
Jul 31 15:30:33.307500 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:33.307268    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"
Jul 31 15:30:33.328510 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:33.328421    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq" podStartSLOduration=3.343849584 podStartE2EDuration="7.328403581s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.984553997s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:30:33.328104465 +0000 UTC m=+2529.639662088" watchObservedRunningTime="2026-07-31 15:30:33.328403581 +0000 UTC m=+2529.639961203"
Jul 31 15:30:34.310995 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:34.310959    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"
Jul 31 15:30:34.312278 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:34.312253    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq" podUID="c01b0f83-8f47-4505-8887-181df5342228" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 31 15:30:35.315167 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:35.315123    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq" podUID="c01b0f83-8f47-4505-8887-181df5342228" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 31 15:30:40.319491 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:40.319451    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"
Jul 31 15:30:40.320089 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:40.319934    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq" podUID="c01b0f83-8f47-4505-8887-181df5342228" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 31 15:30:50.320039 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:30:50.320001    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq" podUID="c01b0f83-8f47-4505-8887-181df5342228" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 31 15:31:00.320107 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:00.320061    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq" podUID="c01b0f83-8f47-4505-8887-181df5342228" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 31 15:31:10.320526 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:10.320470    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq" podUID="c01b0f83-8f47-4505-8887-181df5342228" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 31 15:31:20.320827 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:20.320786    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq" podUID="c01b0f83-8f47-4505-8887-181df5342228" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 31 15:31:28.875126 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:28.875090    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"]
Jul 31 15:31:28.875506 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:28.875365    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68ef276-e099-43e3-b088-fe194f57f0ca" containerName="kube-rbac-proxy"
Jul 31 15:31:28.875506 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:28.875376    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68ef276-e099-43e3-b088-fe194f57f0ca" containerName="kube-rbac-proxy"
Jul 31 15:31:28.875506 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:28.875390    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68ef276-e099-43e3-b088-fe194f57f0ca" containerName="storage-initializer"
Jul 31 15:31:28.875506 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:28.875395    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68ef276-e099-43e3-b088-fe194f57f0ca" containerName="storage-initializer"
Jul 31 15:31:28.875506 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:28.875407    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68ef276-e099-43e3-b088-fe194f57f0ca" containerName="kserve-container"
Jul 31 15:31:28.875506 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:28.875412    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68ef276-e099-43e3-b088-fe194f57f0ca" containerName="kserve-container"
Jul 31 15:31:28.875506 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:28.875461    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a68ef276-e099-43e3-b088-fe194f57f0ca" containerName="kserve-container"
Jul 31 15:31:28.875506 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:28.875473    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a68ef276-e099-43e3-b088-fe194f57f0ca" containerName="kube-rbac-proxy"
Jul 31 15:31:28.878583 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:28.878566    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"
Jul 31 15:31:28.885226 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:28.885206    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Jul 31 15:31:28.885702 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:28.885684    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 31 15:31:28.906899 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:28.906856    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"]
Jul 31 15:31:28.958187 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:28.958148    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09e9c07d-65e7-4447-97cf-fef2173785b1-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7\" (UID: \"09e9c07d-65e7-4447-97cf-fef2173785b1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"
Jul 31 15:31:28.958187 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:28.958189    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e9c07d-65e7-4447-97cf-fef2173785b1-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7\" (UID: \"09e9c07d-65e7-4447-97cf-fef2173785b1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"
Jul 31 15:31:28.958414 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:28.958215    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e9c07d-65e7-4447-97cf-fef2173785b1-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7\" (UID: \"09e9c07d-65e7-4447-97cf-fef2173785b1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"
Jul 31 15:31:28.958414 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:28.958273    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgzmv\" (UniqueName: \"kubernetes.io/projected/09e9c07d-65e7-4447-97cf-fef2173785b1-kube-api-access-hgzmv\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7\" (UID: \"09e9c07d-65e7-4447-97cf-fef2173785b1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"
Jul 31 15:31:28.958414 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:28.958301    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09e9c07d-65e7-4447-97cf-fef2173785b1-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7\" (UID: \"09e9c07d-65e7-4447-97cf-fef2173785b1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"
Jul 31 15:31:29.058923 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:29.058880    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e9c07d-65e7-4447-97cf-fef2173785b1-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7\" (UID: \"09e9c07d-65e7-4447-97cf-fef2173785b1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"
Jul 31 15:31:29.058923 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:29.058927    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e9c07d-65e7-4447-97cf-fef2173785b1-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7\" (UID: \"09e9c07d-65e7-4447-97cf-fef2173785b1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"
Jul 31 15:31:29.059170 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:29.058951    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgzmv\" (UniqueName: \"kubernetes.io/projected/09e9c07d-65e7-4447-97cf-fef2173785b1-kube-api-access-hgzmv\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7\" (UID: \"09e9c07d-65e7-4447-97cf-fef2173785b1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"
Jul 31 15:31:29.059170 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:29.058971    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09e9c07d-65e7-4447-97cf-fef2173785b1-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7\" (UID: \"09e9c07d-65e7-4447-97cf-fef2173785b1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"
Jul 31 15:31:29.059286 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:29.059182    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09e9c07d-65e7-4447-97cf-fef2173785b1-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7\" (UID: \"09e9c07d-65e7-4447-97cf-fef2173785b1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"
Jul 31 15:31:29.059392 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:29.059374    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09e9c07d-65e7-4447-97cf-fef2173785b1-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7\" (UID: \"09e9c07d-65e7-4447-97cf-fef2173785b1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"
Jul 31 15:31:29.059683 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:29.059661    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e9c07d-65e7-4447-97cf-fef2173785b1-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7\" (UID: \"09e9c07d-65e7-4447-97cf-fef2173785b1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"
Jul 31 15:31:29.059856 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:29.059837    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09e9c07d-65e7-4447-97cf-fef2173785b1-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7\" (UID: \"09e9c07d-65e7-4447-97cf-fef2173785b1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"
Jul 31 15:31:29.061600 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:29.061575    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e9c07d-65e7-4447-97cf-fef2173785b1-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7\" (UID: \"09e9c07d-65e7-4447-97cf-fef2173785b1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"
Jul 31 15:31:29.069484 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:29.069456    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgzmv\" (UniqueName: \"kubernetes.io/projected/09e9c07d-65e7-4447-97cf-fef2173785b1-kube-api-access-hgzmv\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7\" (UID: \"09e9c07d-65e7-4447-97cf-fef2173785b1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"
Jul 31 15:31:29.188974 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:29.188869    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"
Jul 31 15:31:29.339838 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:29.339801    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"]
Jul 31 15:31:29.342183 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:31:29.342154    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e9c07d_65e7_4447_97cf_fef2173785b1.slice/crio-f123a7a9ad63a45661baec8d34e85ee349b8d914e8759315f7a5d5cd3ffc9407 WatchSource:0}: Error finding container f123a7a9ad63a45661baec8d34e85ee349b8d914e8759315f7a5d5cd3ffc9407: Status 404 returned error can't find the container with id f123a7a9ad63a45661baec8d34e85ee349b8d914e8759315f7a5d5cd3ffc9407
Jul 31 15:31:29.515891 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:29.515800    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7" event={"ID":"09e9c07d-65e7-4447-97cf-fef2173785b1","Type":"ContainerStarted","Data":"7212b0f7b093ce9e8678be1d9cc3491310701fbdda1daa53537c6855b8d2fb90"}
Jul 31 15:31:29.515891 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:29.515860    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7" event={"ID":"09e9c07d-65e7-4447-97cf-fef2173785b1","Type":"ContainerStarted","Data":"f123a7a9ad63a45661baec8d34e85ee349b8d914e8759315f7a5d5cd3ffc9407"}
Jul 31 15:31:30.320615 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:30.320575    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq" podUID="c01b0f83-8f47-4505-8887-181df5342228" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 31 15:31:33.532293 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:33.532263    2559 generic.go:356] "Generic (PLEG): container finished" podID="09e9c07d-65e7-4447-97cf-fef2173785b1" containerID="7212b0f7b093ce9e8678be1d9cc3491310701fbdda1daa53537c6855b8d2fb90" exitCode=0
Jul 31 15:31:33.532650 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:33.532309    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7" event={"ID":"09e9c07d-65e7-4447-97cf-fef2173785b1","Type":"ContainerDied","Data":"7212b0f7b093ce9e8678be1d9cc3491310701fbdda1daa53537c6855b8d2fb90"}
Jul 31 15:31:34.538822 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:34.538778    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7" event={"ID":"09e9c07d-65e7-4447-97cf-fef2173785b1","Type":"ContainerStarted","Data":"5431e424075f53bde462416bab3ca595c055445d938e80189dfa1405017405a7"}
Jul 31 15:31:34.539325 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:34.538842    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7" event={"ID":"09e9c07d-65e7-4447-97cf-fef2173785b1","Type":"ContainerStarted","Data":"b71aae2c0a7ca317529c076dd246a4d44a08e366bf8fdb97288028f301fe5ec7"}
Jul 31 15:31:34.539325 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:34.539125    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"
Jul 31 15:31:34.566282 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:34.566216    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7" podStartSLOduration=2.509994179 podStartE2EDuration="6.566200029s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.05620585s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:31:34.565005577 +0000 UTC m=+2590.876563200" watchObservedRunningTime="2026-07-31 15:31:34.566200029 +0000 UTC m=+2590.877757650"
Jul 31 15:31:35.542792 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:35.542761    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"
Jul 31 15:31:40.320742 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:40.320709    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"
Jul 31 15:31:41.551170 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:41.551140    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"
Jul 31 15:31:46.379690 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:46.379604    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"]
Jul 31 15:31:46.380173 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:46.379869    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq" podUID="c01b0f83-8f47-4505-8887-181df5342228" containerName="kserve-container" containerID="cri-o://344569c3b8ba85b0396d6ec1ca1b3aad74c7f3a6cf9424e8e2fe90c7b87cd222" gracePeriod=30
Jul 31 15:31:46.380173 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:46.379931    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq" podUID="c01b0f83-8f47-4505-8887-181df5342228" containerName="kube-rbac-proxy" containerID="cri-o://3ba2220c4daee892cd352b6ef71bef65d86335464d6a0bdd214a04ec7d01f55b" gracePeriod=30
Jul 31 15:31:46.589503 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:46.589468    2559 generic.go:356] "Generic (PLEG): container finished" podID="c01b0f83-8f47-4505-8887-181df5342228" containerID="3ba2220c4daee892cd352b6ef71bef65d86335464d6a0bdd214a04ec7d01f55b" exitCode=2
Jul 31 15:31:46.589679 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:46.589544    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq" event={"ID":"c01b0f83-8f47-4505-8887-181df5342228","Type":"ContainerDied","Data":"3ba2220c4daee892cd352b6ef71bef65d86335464d6a0bdd214a04ec7d01f55b"}
Jul 31 15:31:50.021998 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.021970    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"
Jul 31 15:31:50.108949 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.108921    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c01b0f83-8f47-4505-8887-181df5342228-proxy-tls\") pod \"c01b0f83-8f47-4505-8887-181df5342228\" (UID: \"c01b0f83-8f47-4505-8887-181df5342228\") "
Jul 31 15:31:50.109150 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.108969    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsr65\" (UniqueName: \"kubernetes.io/projected/c01b0f83-8f47-4505-8887-181df5342228-kube-api-access-jsr65\") pod \"c01b0f83-8f47-4505-8887-181df5342228\" (UID: \"c01b0f83-8f47-4505-8887-181df5342228\") "
Jul 31 15:31:50.109150 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.109016    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c01b0f83-8f47-4505-8887-181df5342228-kube-rbac-proxy-sar-config\") pod \"c01b0f83-8f47-4505-8887-181df5342228\" (UID: \"c01b0f83-8f47-4505-8887-181df5342228\") "
Jul 31 15:31:50.109150 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.109052    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c01b0f83-8f47-4505-8887-181df5342228-kserve-provision-location\") pod \"c01b0f83-8f47-4505-8887-181df5342228\" (UID: \"c01b0f83-8f47-4505-8887-181df5342228\") "
Jul 31 15:31:50.109150 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.109077    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c01b0f83-8f47-4505-8887-181df5342228-cabundle-cert\") pod \"c01b0f83-8f47-4505-8887-181df5342228\" (UID: \"c01b0f83-8f47-4505-8887-181df5342228\") "
Jul 31 15:31:50.109476 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.109445    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c01b0f83-8f47-4505-8887-181df5342228-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c01b0f83-8f47-4505-8887-181df5342228" (UID: "c01b0f83-8f47-4505-8887-181df5342228"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:31:50.109721 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.109507    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c01b0f83-8f47-4505-8887-181df5342228-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c01b0f83-8f47-4505-8887-181df5342228" (UID: "c01b0f83-8f47-4505-8887-181df5342228"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:31:50.109827 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.109806    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01b0f83-8f47-4505-8887-181df5342228-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c01b0f83-8f47-4505-8887-181df5342228" (UID: "c01b0f83-8f47-4505-8887-181df5342228"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:31:50.111099 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.111072    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c01b0f83-8f47-4505-8887-181df5342228-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c01b0f83-8f47-4505-8887-181df5342228" (UID: "c01b0f83-8f47-4505-8887-181df5342228"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:31:50.111234 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.111210    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01b0f83-8f47-4505-8887-181df5342228-kube-api-access-jsr65" (OuterVolumeSpecName: "kube-api-access-jsr65") pod "c01b0f83-8f47-4505-8887-181df5342228" (UID: "c01b0f83-8f47-4505-8887-181df5342228"). InnerVolumeSpecName "kube-api-access-jsr65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:31:50.210013 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.209923    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c01b0f83-8f47-4505-8887-181df5342228-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:31:50.210013 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.209953    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c01b0f83-8f47-4505-8887-181df5342228-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:31:50.210013 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.209964    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c01b0f83-8f47-4505-8887-181df5342228-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:31:50.210013 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.209972    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsr65\" (UniqueName: \"kubernetes.io/projected/c01b0f83-8f47-4505-8887-181df5342228-kube-api-access-jsr65\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:31:50.210013 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.209981    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c01b0f83-8f47-4505-8887-181df5342228-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:31:50.604953 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.604924    2559 generic.go:356] "Generic (PLEG): container finished" podID="c01b0f83-8f47-4505-8887-181df5342228" containerID="344569c3b8ba85b0396d6ec1ca1b3aad74c7f3a6cf9424e8e2fe90c7b87cd222" exitCode=0
Jul 31 15:31:50.605134 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.604975    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"
Jul 31 15:31:50.605134 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.605009    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq" event={"ID":"c01b0f83-8f47-4505-8887-181df5342228","Type":"ContainerDied","Data":"344569c3b8ba85b0396d6ec1ca1b3aad74c7f3a6cf9424e8e2fe90c7b87cd222"}
Jul 31 15:31:50.605134 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.605063    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq" event={"ID":"c01b0f83-8f47-4505-8887-181df5342228","Type":"ContainerDied","Data":"b632076860beac119e120b25d85f4605e017109d46575860f77823c87559a1b4"}
Jul 31 15:31:50.605134 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.605097    2559 scope.go:117] "RemoveContainer" containerID="3ba2220c4daee892cd352b6ef71bef65d86335464d6a0bdd214a04ec7d01f55b"
Jul 31 15:31:50.613143 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.613093    2559 scope.go:117] "RemoveContainer" containerID="344569c3b8ba85b0396d6ec1ca1b3aad74c7f3a6cf9424e8e2fe90c7b87cd222"
Jul 31 15:31:50.620299 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.620280    2559 scope.go:117] "RemoveContainer" containerID="ca83d7af48f1c4155a2a575d128a89ede52148a17e0c063b0a9e7efd4bbab0ec"
Jul 31 15:31:50.625186 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.625164    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"]
Jul 31 15:31:50.628104 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.628084    2559 scope.go:117] "RemoveContainer" containerID="3ba2220c4daee892cd352b6ef71bef65d86335464d6a0bdd214a04ec7d01f55b"
Jul 31 15:31:50.628642 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:31:50.628620    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ba2220c4daee892cd352b6ef71bef65d86335464d6a0bdd214a04ec7d01f55b\": container with ID starting with 3ba2220c4daee892cd352b6ef71bef65d86335464d6a0bdd214a04ec7d01f55b not found: ID does not exist" containerID="3ba2220c4daee892cd352b6ef71bef65d86335464d6a0bdd214a04ec7d01f55b"
Jul 31 15:31:50.628792 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.628648    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ba2220c4daee892cd352b6ef71bef65d86335464d6a0bdd214a04ec7d01f55b"} err="failed to get container status \"3ba2220c4daee892cd352b6ef71bef65d86335464d6a0bdd214a04ec7d01f55b\": rpc error: code = NotFound desc = could not find container \"3ba2220c4daee892cd352b6ef71bef65d86335464d6a0bdd214a04ec7d01f55b\": container with ID starting with 3ba2220c4daee892cd352b6ef71bef65d86335464d6a0bdd214a04ec7d01f55b not found: ID does not exist"
Jul 31 15:31:50.628792 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.628667    2559 scope.go:117] "RemoveContainer" containerID="344569c3b8ba85b0396d6ec1ca1b3aad74c7f3a6cf9424e8e2fe90c7b87cd222"
Jul 31 15:31:50.629013 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:31:50.628991    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"344569c3b8ba85b0396d6ec1ca1b3aad74c7f3a6cf9424e8e2fe90c7b87cd222\": container with ID starting with 344569c3b8ba85b0396d6ec1ca1b3aad74c7f3a6cf9424e8e2fe90c7b87cd222 not found: ID does not exist" containerID="344569c3b8ba85b0396d6ec1ca1b3aad74c7f3a6cf9424e8e2fe90c7b87cd222"
Jul 31 15:31:50.629073 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.629025    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"344569c3b8ba85b0396d6ec1ca1b3aad74c7f3a6cf9424e8e2fe90c7b87cd222"} err="failed to get container status \"344569c3b8ba85b0396d6ec1ca1b3aad74c7f3a6cf9424e8e2fe90c7b87cd222\": rpc error: code = NotFound desc = could not find container \"344569c3b8ba85b0396d6ec1ca1b3aad74c7f3a6cf9424e8e2fe90c7b87cd222\": container with ID starting with 344569c3b8ba85b0396d6ec1ca1b3aad74c7f3a6cf9424e8e2fe90c7b87cd222 not found: ID does not exist"
Jul 31 15:31:50.629073 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.629045    2559 scope.go:117] "RemoveContainer" containerID="ca83d7af48f1c4155a2a575d128a89ede52148a17e0c063b0a9e7efd4bbab0ec"
Jul 31 15:31:50.629298 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:31:50.629279    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca83d7af48f1c4155a2a575d128a89ede52148a17e0c063b0a9e7efd4bbab0ec\": container with ID starting with ca83d7af48f1c4155a2a575d128a89ede52148a17e0c063b0a9e7efd4bbab0ec not found: ID does not exist" containerID="ca83d7af48f1c4155a2a575d128a89ede52148a17e0c063b0a9e7efd4bbab0ec"
Jul 31 15:31:50.629357 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.629303    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca83d7af48f1c4155a2a575d128a89ede52148a17e0c063b0a9e7efd4bbab0ec"} err="failed to get container status \"ca83d7af48f1c4155a2a575d128a89ede52148a17e0c063b0a9e7efd4bbab0ec\": rpc error: code = NotFound desc = could not find container \"ca83d7af48f1c4155a2a575d128a89ede52148a17e0c063b0a9e7efd4bbab0ec\": container with ID starting with ca83d7af48f1c4155a2a575d128a89ede52148a17e0c063b0a9e7efd4bbab0ec not found: ID does not exist"
Jul 31 15:31:50.629795 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:50.629777    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-dctqq"]
Jul 31 15:31:51.546131 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.546095    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"]
Jul 31 15:31:51.546526 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.546362    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c01b0f83-8f47-4505-8887-181df5342228" containerName="kserve-container"
Jul 31 15:31:51.546526 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.546373    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c01b0f83-8f47-4505-8887-181df5342228" containerName="kserve-container"
Jul 31 15:31:51.546526 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.546392    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c01b0f83-8f47-4505-8887-181df5342228" containerName="kube-rbac-proxy"
Jul 31 15:31:51.546526 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.546399    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c01b0f83-8f47-4505-8887-181df5342228" containerName="kube-rbac-proxy"
Jul 31 15:31:51.546526 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.546413    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c01b0f83-8f47-4505-8887-181df5342228" containerName="storage-initializer"
Jul 31 15:31:51.546526 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.546422    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c01b0f83-8f47-4505-8887-181df5342228" containerName="storage-initializer"
Jul 31 15:31:51.546526 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.546485    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c01b0f83-8f47-4505-8887-181df5342228" containerName="kserve-container"
Jul 31 15:31:51.546526 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.546498    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c01b0f83-8f47-4505-8887-181df5342228" containerName="kube-rbac-proxy"
Jul 31 15:31:51.551141 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.551119    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"
Jul 31 15:31:51.553568 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.553549    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 31 15:31:51.553797 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.553778    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Jul 31 15:31:51.561756 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.561719    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"]
Jul 31 15:31:51.618932 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.618898    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7737771-79a6-4951-a4eb-09b5316f0ee5-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq\" (UID: \"d7737771-79a6-4951-a4eb-09b5316f0ee5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"
Jul 31 15:31:51.619094 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.618938    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7737771-79a6-4951-a4eb-09b5316f0ee5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq\" (UID: \"d7737771-79a6-4951-a4eb-09b5316f0ee5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"
Jul 31 15:31:51.619094 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.619021    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xhhg\" (UniqueName: \"kubernetes.io/projected/d7737771-79a6-4951-a4eb-09b5316f0ee5-kube-api-access-7xhhg\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq\" (UID: \"d7737771-79a6-4951-a4eb-09b5316f0ee5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"
Jul 31 15:31:51.619094 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.619053    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7737771-79a6-4951-a4eb-09b5316f0ee5-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq\" (UID: \"d7737771-79a6-4951-a4eb-09b5316f0ee5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"
Jul 31 15:31:51.619094 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.619091    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7737771-79a6-4951-a4eb-09b5316f0ee5-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq\" (UID: \"d7737771-79a6-4951-a4eb-09b5316f0ee5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"
Jul 31 15:31:51.719572 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.719534    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7737771-79a6-4951-a4eb-09b5316f0ee5-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq\" (UID: \"d7737771-79a6-4951-a4eb-09b5316f0ee5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"
Jul 31 15:31:51.719790 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.719585    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7737771-79a6-4951-a4eb-09b5316f0ee5-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq\" (UID: \"d7737771-79a6-4951-a4eb-09b5316f0ee5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"
Jul 31 15:31:51.719790 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.719710    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7737771-79a6-4951-a4eb-09b5316f0ee5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq\" (UID: \"d7737771-79a6-4951-a4eb-09b5316f0ee5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"
Jul 31 15:31:51.719908 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.719817    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xhhg\" (UniqueName: \"kubernetes.io/projected/d7737771-79a6-4951-a4eb-09b5316f0ee5-kube-api-access-7xhhg\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq\" (UID: \"d7737771-79a6-4951-a4eb-09b5316f0ee5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"
Jul 31 15:31:51.719908 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.719856    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7737771-79a6-4951-a4eb-09b5316f0ee5-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq\" (UID: \"d7737771-79a6-4951-a4eb-09b5316f0ee5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"
Jul 31 15:31:51.720049 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.720027    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7737771-79a6-4951-a4eb-09b5316f0ee5-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq\" (UID: \"d7737771-79a6-4951-a4eb-09b5316f0ee5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"
Jul 31 15:31:51.720393 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.720366    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7737771-79a6-4951-a4eb-09b5316f0ee5-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq\" (UID: \"d7737771-79a6-4951-a4eb-09b5316f0ee5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"
Jul 31 15:31:51.720509 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.720447    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7737771-79a6-4951-a4eb-09b5316f0ee5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq\" (UID: \"d7737771-79a6-4951-a4eb-09b5316f0ee5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"
Jul 31 15:31:51.722520 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.722501    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7737771-79a6-4951-a4eb-09b5316f0ee5-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq\" (UID: \"d7737771-79a6-4951-a4eb-09b5316f0ee5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"
Jul 31 15:31:51.728691 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.728659    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xhhg\" (UniqueName: \"kubernetes.io/projected/d7737771-79a6-4951-a4eb-09b5316f0ee5-kube-api-access-7xhhg\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq\" (UID: \"d7737771-79a6-4951-a4eb-09b5316f0ee5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"
Jul 31 15:31:51.862170 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.862141    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"
Jul 31 15:31:51.996254 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:51.996226    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"]
Jul 31 15:31:51.998309 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:31:51.998281    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7737771_79a6_4951_a4eb_09b5316f0ee5.slice/crio-59ec8abd4649a87d56f85e2dac0097b437a4e9a4f9c9be7b30a2406b7af0fbff WatchSource:0}: Error finding container 59ec8abd4649a87d56f85e2dac0097b437a4e9a4f9c9be7b30a2406b7af0fbff: Status 404 returned error can't find the container with id 59ec8abd4649a87d56f85e2dac0097b437a4e9a4f9c9be7b30a2406b7af0fbff
Jul 31 15:31:52.220890 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:52.220809    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c01b0f83-8f47-4505-8887-181df5342228" path="/var/lib/kubelet/pods/c01b0f83-8f47-4505-8887-181df5342228/volumes"
Jul 31 15:31:52.616039 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:52.616000    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq" event={"ID":"d7737771-79a6-4951-a4eb-09b5316f0ee5","Type":"ContainerStarted","Data":"f5d4bb331773a321074adcd8e2bc2e0d57e328849f4e692581959ee65d0b1fbe"}
Jul 31 15:31:52.616429 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:52.616075    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq" event={"ID":"d7737771-79a6-4951-a4eb-09b5316f0ee5","Type":"ContainerStarted","Data":"59ec8abd4649a87d56f85e2dac0097b437a4e9a4f9c9be7b30a2406b7af0fbff"}
Jul 31 15:31:56.635262 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:56.635230    2559 generic.go:356] "Generic (PLEG): container finished" podID="d7737771-79a6-4951-a4eb-09b5316f0ee5" containerID="f5d4bb331773a321074adcd8e2bc2e0d57e328849f4e692581959ee65d0b1fbe" exitCode=0
Jul 31 15:31:56.635629 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:56.635282    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq" event={"ID":"d7737771-79a6-4951-a4eb-09b5316f0ee5","Type":"ContainerDied","Data":"f5d4bb331773a321074adcd8e2bc2e0d57e328849f4e692581959ee65d0b1fbe"}
Jul 31 15:31:57.641783 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:57.641721    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq" event={"ID":"d7737771-79a6-4951-a4eb-09b5316f0ee5","Type":"ContainerStarted","Data":"770b1550524740e3e6b2a4c2c01484df1699caedeea41c78d5d3d05898aeb143"}
Jul 31 15:31:57.642160 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:57.641828    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq" event={"ID":"d7737771-79a6-4951-a4eb-09b5316f0ee5","Type":"ContainerStarted","Data":"74c5d11f45743fd550d528043552470b861763b85cd0125a2096003081730e41"}
Jul 31 15:31:57.642160 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:57.642061    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"
Jul 31 15:31:57.671767 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:57.668804    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq" podStartSLOduration=2.973063245 podStartE2EDuration="6.66878531s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.695722065s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:31:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:31:57.66618735 +0000 UTC m=+2613.977744973" watchObservedRunningTime="2026-07-31 15:31:57.66878531 +0000 UTC m=+2613.980342936"
Jul 31 15:31:58.645800 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:31:58.645764    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"
Jul 31 15:32:04.654614 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:04.654586    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"
Jul 31 15:32:11.554846 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:11.554808    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"
Jul 31 15:32:18.923747 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:18.923698    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"]
Jul 31 15:32:18.924188 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:18.924015    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7" podUID="09e9c07d-65e7-4447-97cf-fef2173785b1" containerName="kserve-container" containerID="cri-o://b71aae2c0a7ca317529c076dd246a4d44a08e366bf8fdb97288028f301fe5ec7" gracePeriod=30
Jul 31 15:32:18.924188 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:18.924039    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7" podUID="09e9c07d-65e7-4447-97cf-fef2173785b1" containerName="kube-rbac-proxy" containerID="cri-o://5431e424075f53bde462416bab3ca595c055445d938e80189dfa1405017405a7" gracePeriod=30
Jul 31 15:32:19.722930 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:19.722900    2559 generic.go:356] "Generic (PLEG): container finished" podID="09e9c07d-65e7-4447-97cf-fef2173785b1" containerID="5431e424075f53bde462416bab3ca595c055445d938e80189dfa1405017405a7" exitCode=2
Jul 31 15:32:19.723117 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:19.722955    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7" event={"ID":"09e9c07d-65e7-4447-97cf-fef2173785b1","Type":"ContainerDied","Data":"5431e424075f53bde462416bab3ca595c055445d938e80189dfa1405017405a7"}
Jul 31 15:32:20.066025 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.065994    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"
Jul 31 15:32:20.146647 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.146617    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09e9c07d-65e7-4447-97cf-fef2173785b1-cabundle-cert\") pod \"09e9c07d-65e7-4447-97cf-fef2173785b1\" (UID: \"09e9c07d-65e7-4447-97cf-fef2173785b1\") "
Jul 31 15:32:20.146844 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.146654    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgzmv\" (UniqueName: \"kubernetes.io/projected/09e9c07d-65e7-4447-97cf-fef2173785b1-kube-api-access-hgzmv\") pod \"09e9c07d-65e7-4447-97cf-fef2173785b1\" (UID: \"09e9c07d-65e7-4447-97cf-fef2173785b1\") "
Jul 31 15:32:20.146844 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.146718    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e9c07d-65e7-4447-97cf-fef2173785b1-proxy-tls\") pod \"09e9c07d-65e7-4447-97cf-fef2173785b1\" (UID: \"09e9c07d-65e7-4447-97cf-fef2173785b1\") "
Jul 31 15:32:20.146972 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.146887    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09e9c07d-65e7-4447-97cf-fef2173785b1-kserve-provision-location\") pod \"09e9c07d-65e7-4447-97cf-fef2173785b1\" (UID: \"09e9c07d-65e7-4447-97cf-fef2173785b1\") "
Jul 31 15:32:20.146972 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.146931    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e9c07d-65e7-4447-97cf-fef2173785b1-kube-rbac-proxy-sar-config\") pod \"09e9c07d-65e7-4447-97cf-fef2173785b1\" (UID: \"09e9c07d-65e7-4447-97cf-fef2173785b1\") "
Jul 31 15:32:20.147098 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.147074    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e9c07d-65e7-4447-97cf-fef2173785b1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "09e9c07d-65e7-4447-97cf-fef2173785b1" (UID: "09e9c07d-65e7-4447-97cf-fef2173785b1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:32:20.147200 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.147167    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09e9c07d-65e7-4447-97cf-fef2173785b1-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:32:20.147330 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.147249    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09e9c07d-65e7-4447-97cf-fef2173785b1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "09e9c07d-65e7-4447-97cf-fef2173785b1" (UID: "09e9c07d-65e7-4447-97cf-fef2173785b1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:32:20.147330 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.147291    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e9c07d-65e7-4447-97cf-fef2173785b1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "09e9c07d-65e7-4447-97cf-fef2173785b1" (UID: "09e9c07d-65e7-4447-97cf-fef2173785b1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:32:20.149411 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.149385    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09e9c07d-65e7-4447-97cf-fef2173785b1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "09e9c07d-65e7-4447-97cf-fef2173785b1" (UID: "09e9c07d-65e7-4447-97cf-fef2173785b1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:32:20.149411 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.149381    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e9c07d-65e7-4447-97cf-fef2173785b1-kube-api-access-hgzmv" (OuterVolumeSpecName: "kube-api-access-hgzmv") pod "09e9c07d-65e7-4447-97cf-fef2173785b1" (UID: "09e9c07d-65e7-4447-97cf-fef2173785b1"). InnerVolumeSpecName "kube-api-access-hgzmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:32:20.248621 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.248546    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e9c07d-65e7-4447-97cf-fef2173785b1-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:32:20.248621 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.248579    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09e9c07d-65e7-4447-97cf-fef2173785b1-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:32:20.248621 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.248592    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e9c07d-65e7-4447-97cf-fef2173785b1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:32:20.248621 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.248600    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgzmv\" (UniqueName: \"kubernetes.io/projected/09e9c07d-65e7-4447-97cf-fef2173785b1-kube-api-access-hgzmv\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:32:20.728862 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.728826    2559 generic.go:356] "Generic (PLEG): container finished" podID="09e9c07d-65e7-4447-97cf-fef2173785b1" containerID="b71aae2c0a7ca317529c076dd246a4d44a08e366bf8fdb97288028f301fe5ec7" exitCode=0
Jul 31 15:32:20.729035 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.728890    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"
Jul 31 15:32:20.729035 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.728906    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7" event={"ID":"09e9c07d-65e7-4447-97cf-fef2173785b1","Type":"ContainerDied","Data":"b71aae2c0a7ca317529c076dd246a4d44a08e366bf8fdb97288028f301fe5ec7"}
Jul 31 15:32:20.729035 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.728969    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7" event={"ID":"09e9c07d-65e7-4447-97cf-fef2173785b1","Type":"ContainerDied","Data":"f123a7a9ad63a45661baec8d34e85ee349b8d914e8759315f7a5d5cd3ffc9407"}
Jul 31 15:32:20.729035 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.728995    2559 scope.go:117] "RemoveContainer" containerID="5431e424075f53bde462416bab3ca595c055445d938e80189dfa1405017405a7"
Jul 31 15:32:20.737440 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.737419    2559 scope.go:117] "RemoveContainer" containerID="b71aae2c0a7ca317529c076dd246a4d44a08e366bf8fdb97288028f301fe5ec7"
Jul 31 15:32:20.747220 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.747195    2559 scope.go:117] "RemoveContainer" containerID="7212b0f7b093ce9e8678be1d9cc3491310701fbdda1daa53537c6855b8d2fb90"
Jul 31 15:32:20.754906 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.754884    2559 scope.go:117] "RemoveContainer" containerID="5431e424075f53bde462416bab3ca595c055445d938e80189dfa1405017405a7"
Jul 31 15:32:20.755208 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:32:20.755189    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5431e424075f53bde462416bab3ca595c055445d938e80189dfa1405017405a7\": container with ID starting with 5431e424075f53bde462416bab3ca595c055445d938e80189dfa1405017405a7 not found: ID does not exist" containerID="5431e424075f53bde462416bab3ca595c055445d938e80189dfa1405017405a7"
Jul 31 15:32:20.755289 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.755223    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5431e424075f53bde462416bab3ca595c055445d938e80189dfa1405017405a7"} err="failed to get container status \"5431e424075f53bde462416bab3ca595c055445d938e80189dfa1405017405a7\": rpc error: code = NotFound desc = could not find container \"5431e424075f53bde462416bab3ca595c055445d938e80189dfa1405017405a7\": container with ID starting with 5431e424075f53bde462416bab3ca595c055445d938e80189dfa1405017405a7 not found: ID does not exist"
Jul 31 15:32:20.755289 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.755250    2559 scope.go:117] "RemoveContainer" containerID="b71aae2c0a7ca317529c076dd246a4d44a08e366bf8fdb97288028f301fe5ec7"
Jul 31 15:32:20.755496 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:32:20.755478    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b71aae2c0a7ca317529c076dd246a4d44a08e366bf8fdb97288028f301fe5ec7\": container with ID starting with b71aae2c0a7ca317529c076dd246a4d44a08e366bf8fdb97288028f301fe5ec7 not found: ID does not exist" containerID="b71aae2c0a7ca317529c076dd246a4d44a08e366bf8fdb97288028f301fe5ec7"
Jul 31 15:32:20.755534 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.755502    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b71aae2c0a7ca317529c076dd246a4d44a08e366bf8fdb97288028f301fe5ec7"} err="failed to get container status \"b71aae2c0a7ca317529c076dd246a4d44a08e366bf8fdb97288028f301fe5ec7\": rpc error: code = NotFound desc = could not find container \"b71aae2c0a7ca317529c076dd246a4d44a08e366bf8fdb97288028f301fe5ec7\": container with ID starting with b71aae2c0a7ca317529c076dd246a4d44a08e366bf8fdb97288028f301fe5ec7 not found: ID does not exist"
Jul 31 15:32:20.755534 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.755517    2559 scope.go:117] "RemoveContainer" containerID="7212b0f7b093ce9e8678be1d9cc3491310701fbdda1daa53537c6855b8d2fb90"
Jul 31 15:32:20.755819 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:32:20.755795    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7212b0f7b093ce9e8678be1d9cc3491310701fbdda1daa53537c6855b8d2fb90\": container with ID starting with 7212b0f7b093ce9e8678be1d9cc3491310701fbdda1daa53537c6855b8d2fb90 not found: ID does not exist" containerID="7212b0f7b093ce9e8678be1d9cc3491310701fbdda1daa53537c6855b8d2fb90"
Jul 31 15:32:20.755918 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.755821    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7212b0f7b093ce9e8678be1d9cc3491310701fbdda1daa53537c6855b8d2fb90"} err="failed to get container status \"7212b0f7b093ce9e8678be1d9cc3491310701fbdda1daa53537c6855b8d2fb90\": rpc error: code = NotFound desc = could not find container \"7212b0f7b093ce9e8678be1d9cc3491310701fbdda1daa53537c6855b8d2fb90\": container with ID starting with 7212b0f7b093ce9e8678be1d9cc3491310701fbdda1daa53537c6855b8d2fb90 not found: ID does not exist"
Jul 31 15:32:20.758456 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.758431    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"]
Jul 31 15:32:20.767065 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:20.767038    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-g9cr7"]
Jul 31 15:32:22.221135 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:22.221105    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09e9c07d-65e7-4447-97cf-fef2173785b1" path="/var/lib/kubelet/pods/09e9c07d-65e7-4447-97cf-fef2173785b1/volumes"
Jul 31 15:32:34.665328 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:34.665285    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq" podUID="d7737771-79a6-4951-a4eb-09b5316f0ee5" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 31 15:32:44.657219 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:44.657190    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"
Jul 31 15:32:51.614103 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:51.614072    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"]
Jul 31 15:32:51.614569 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:51.614286    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq" podUID="d7737771-79a6-4951-a4eb-09b5316f0ee5" containerName="kserve-container" containerID="cri-o://74c5d11f45743fd550d528043552470b861763b85cd0125a2096003081730e41" gracePeriod=30
Jul 31 15:32:51.614569 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:51.614340    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq" podUID="d7737771-79a6-4951-a4eb-09b5316f0ee5" containerName="kube-rbac-proxy" containerID="cri-o://770b1550524740e3e6b2a4c2c01484df1699caedeea41c78d5d3d05898aeb143" gracePeriod=30
Jul 31 15:32:51.852669 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:51.852636    2559 generic.go:356] "Generic (PLEG): container finished" podID="d7737771-79a6-4951-a4eb-09b5316f0ee5" containerID="770b1550524740e3e6b2a4c2c01484df1699caedeea41c78d5d3d05898aeb143" exitCode=2
Jul 31 15:32:51.852844 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:51.852709    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq" event={"ID":"d7737771-79a6-4951-a4eb-09b5316f0ee5","Type":"ContainerDied","Data":"770b1550524740e3e6b2a4c2c01484df1699caedeea41c78d5d3d05898aeb143"}
Jul 31 15:32:54.650066 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:54.650020    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq" podUID="d7737771-79a6-4951-a4eb-09b5316f0ee5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Jul 31 15:32:59.157867 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.157840    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"
Jul 31 15:32:59.206625 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.206591    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"]
Jul 31 15:32:59.206913 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.206900    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7737771-79a6-4951-a4eb-09b5316f0ee5" containerName="kube-rbac-proxy"
Jul 31 15:32:59.206964 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.206916    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7737771-79a6-4951-a4eb-09b5316f0ee5" containerName="kube-rbac-proxy"
Jul 31 15:32:59.206964 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.206926    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09e9c07d-65e7-4447-97cf-fef2173785b1" containerName="kserve-container"
Jul 31 15:32:59.206964 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.206932    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="09e9c07d-65e7-4447-97cf-fef2173785b1" containerName="kserve-container"
Jul 31 15:32:59.206964 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.206942    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09e9c07d-65e7-4447-97cf-fef2173785b1" containerName="kube-rbac-proxy"
Jul 31 15:32:59.206964 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.206948    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="09e9c07d-65e7-4447-97cf-fef2173785b1" containerName="kube-rbac-proxy"
Jul 31 15:32:59.206964 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.206959    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09e9c07d-65e7-4447-97cf-fef2173785b1" containerName="storage-initializer"
Jul 31 15:32:59.206964 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.206964    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="09e9c07d-65e7-4447-97cf-fef2173785b1" containerName="storage-initializer"
Jul 31 15:32:59.207179 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.206973    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7737771-79a6-4951-a4eb-09b5316f0ee5" containerName="storage-initializer"
Jul 31 15:32:59.207179 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.206980    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7737771-79a6-4951-a4eb-09b5316f0ee5" containerName="storage-initializer"
Jul 31 15:32:59.207179 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.206987    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7737771-79a6-4951-a4eb-09b5316f0ee5" containerName="kserve-container"
Jul 31 15:32:59.207179 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.206992    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7737771-79a6-4951-a4eb-09b5316f0ee5" containerName="kserve-container"
Jul 31 15:32:59.207179 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.207037    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="09e9c07d-65e7-4447-97cf-fef2173785b1" containerName="kube-rbac-proxy"
Jul 31 15:32:59.207179 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.207043    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="09e9c07d-65e7-4447-97cf-fef2173785b1" containerName="kserve-container"
Jul 31 15:32:59.207179 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.207049    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7737771-79a6-4951-a4eb-09b5316f0ee5" containerName="kserve-container"
Jul 31 15:32:59.207179 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.207054    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7737771-79a6-4951-a4eb-09b5316f0ee5" containerName="kube-rbac-proxy"
Jul 31 15:32:59.209953 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.209933    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"
Jul 31 15:32:59.212274 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.212255    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 31 15:32:59.212274 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.212271    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Jul 31 15:32:59.220079 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.220058    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"]
Jul 31 15:32:59.251747 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.249272    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d091f55f-2add-44b7-9f9f-e3ef87ae543c-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b\" (UID: \"d091f55f-2add-44b7-9f9f-e3ef87ae543c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"
Jul 31 15:32:59.251747 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.249361    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d091f55f-2add-44b7-9f9f-e3ef87ae543c-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b\" (UID: \"d091f55f-2add-44b7-9f9f-e3ef87ae543c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"
Jul 31 15:32:59.251747 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.249418    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpgc8\" (UniqueName: \"kubernetes.io/projected/d091f55f-2add-44b7-9f9f-e3ef87ae543c-kube-api-access-mpgc8\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b\" (UID: \"d091f55f-2add-44b7-9f9f-e3ef87ae543c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"
Jul 31 15:32:59.251747 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.249456    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d091f55f-2add-44b7-9f9f-e3ef87ae543c-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b\" (UID: \"d091f55f-2add-44b7-9f9f-e3ef87ae543c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"
Jul 31 15:32:59.251747 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.249534    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d091f55f-2add-44b7-9f9f-e3ef87ae543c-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b\" (UID: \"d091f55f-2add-44b7-9f9f-e3ef87ae543c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"
Jul 31 15:32:59.350815 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.350781    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7737771-79a6-4951-a4eb-09b5316f0ee5-cabundle-cert\") pod \"d7737771-79a6-4951-a4eb-09b5316f0ee5\" (UID: \"d7737771-79a6-4951-a4eb-09b5316f0ee5\") "
Jul 31 15:32:59.351027 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.350830    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7737771-79a6-4951-a4eb-09b5316f0ee5-kube-rbac-proxy-sar-config\") pod \"d7737771-79a6-4951-a4eb-09b5316f0ee5\" (UID: \"d7737771-79a6-4951-a4eb-09b5316f0ee5\") "
Jul 31 15:32:59.351027 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.350863    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7737771-79a6-4951-a4eb-09b5316f0ee5-kserve-provision-location\") pod \"d7737771-79a6-4951-a4eb-09b5316f0ee5\" (UID: \"d7737771-79a6-4951-a4eb-09b5316f0ee5\") "
Jul 31 15:32:59.351027 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.350881    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7737771-79a6-4951-a4eb-09b5316f0ee5-proxy-tls\") pod \"d7737771-79a6-4951-a4eb-09b5316f0ee5\" (UID: \"d7737771-79a6-4951-a4eb-09b5316f0ee5\") "
Jul 31 15:32:59.351027 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.350906    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xhhg\" (UniqueName: \"kubernetes.io/projected/d7737771-79a6-4951-a4eb-09b5316f0ee5-kube-api-access-7xhhg\") pod \"d7737771-79a6-4951-a4eb-09b5316f0ee5\" (UID: \"d7737771-79a6-4951-a4eb-09b5316f0ee5\") "
Jul 31 15:32:59.351027 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.351017    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpgc8\" (UniqueName: \"kubernetes.io/projected/d091f55f-2add-44b7-9f9f-e3ef87ae543c-kube-api-access-mpgc8\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b\" (UID: \"d091f55f-2add-44b7-9f9f-e3ef87ae543c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"
Jul 31 15:32:59.351325 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.351055    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d091f55f-2add-44b7-9f9f-e3ef87ae543c-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b\" (UID: \"d091f55f-2add-44b7-9f9f-e3ef87ae543c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"
Jul 31 15:32:59.351325 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.351132    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d091f55f-2add-44b7-9f9f-e3ef87ae543c-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b\" (UID: \"d091f55f-2add-44b7-9f9f-e3ef87ae543c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"
Jul 31 15:32:59.351325 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.351192    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7737771-79a6-4951-a4eb-09b5316f0ee5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d7737771-79a6-4951-a4eb-09b5316f0ee5" (UID: "d7737771-79a6-4951-a4eb-09b5316f0ee5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:32:59.351325 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.351265    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d091f55f-2add-44b7-9f9f-e3ef87ae543c-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b\" (UID: \"d091f55f-2add-44b7-9f9f-e3ef87ae543c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"
Jul 31 15:32:59.351325 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.351293    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d091f55f-2add-44b7-9f9f-e3ef87ae543c-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b\" (UID: \"d091f55f-2add-44b7-9f9f-e3ef87ae543c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"
Jul 31 15:32:59.351325 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.351291    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7737771-79a6-4951-a4eb-09b5316f0ee5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d7737771-79a6-4951-a4eb-09b5316f0ee5" (UID: "d7737771-79a6-4951-a4eb-09b5316f0ee5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:32:59.351325 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.351303    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7737771-79a6-4951-a4eb-09b5316f0ee5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d7737771-79a6-4951-a4eb-09b5316f0ee5" (UID: "d7737771-79a6-4951-a4eb-09b5316f0ee5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:32:59.351688 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.351426    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7737771-79a6-4951-a4eb-09b5316f0ee5-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:32:59.351688 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:32:59.351443    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-serving-cert: secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Jul 31 15:32:59.351688 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:32:59.351606    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d091f55f-2add-44b7-9f9f-e3ef87ae543c-proxy-tls podName:d091f55f-2add-44b7-9f9f-e3ef87ae543c nodeName:}" failed. No retries permitted until 2026-07-31 15:32:59.851581934 +0000 UTC m=+2676.163139551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d091f55f-2add-44b7-9f9f-e3ef87ae543c-proxy-tls") pod "isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b" (UID: "d091f55f-2add-44b7-9f9f-e3ef87ae543c") : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Jul 31 15:32:59.351688 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.351674    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d091f55f-2add-44b7-9f9f-e3ef87ae543c-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b\" (UID: \"d091f55f-2add-44b7-9f9f-e3ef87ae543c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"
Jul 31 15:32:59.352056 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.352035    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d091f55f-2add-44b7-9f9f-e3ef87ae543c-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b\" (UID: \"d091f55f-2add-44b7-9f9f-e3ef87ae543c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"
Jul 31 15:32:59.352162 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.352142    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d091f55f-2add-44b7-9f9f-e3ef87ae543c-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b\" (UID: \"d091f55f-2add-44b7-9f9f-e3ef87ae543c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"
Jul 31 15:32:59.353303 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.353284    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7737771-79a6-4951-a4eb-09b5316f0ee5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d7737771-79a6-4951-a4eb-09b5316f0ee5" (UID: "d7737771-79a6-4951-a4eb-09b5316f0ee5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:32:59.353499 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.353479    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7737771-79a6-4951-a4eb-09b5316f0ee5-kube-api-access-7xhhg" (OuterVolumeSpecName: "kube-api-access-7xhhg") pod "d7737771-79a6-4951-a4eb-09b5316f0ee5" (UID: "d7737771-79a6-4951-a4eb-09b5316f0ee5"). InnerVolumeSpecName "kube-api-access-7xhhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:32:59.360079 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.360057    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpgc8\" (UniqueName: \"kubernetes.io/projected/d091f55f-2add-44b7-9f9f-e3ef87ae543c-kube-api-access-mpgc8\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b\" (UID: \"d091f55f-2add-44b7-9f9f-e3ef87ae543c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"
Jul 31 15:32:59.451917 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.451882    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xhhg\" (UniqueName: \"kubernetes.io/projected/d7737771-79a6-4951-a4eb-09b5316f0ee5-kube-api-access-7xhhg\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:32:59.451917 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.451908    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7737771-79a6-4951-a4eb-09b5316f0ee5-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:32:59.451917 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.451918    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7737771-79a6-4951-a4eb-09b5316f0ee5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:32:59.451917 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.451927    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7737771-79a6-4951-a4eb-09b5316f0ee5-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:32:59.855205 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.855162    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d091f55f-2add-44b7-9f9f-e3ef87ae543c-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b\" (UID: \"d091f55f-2add-44b7-9f9f-e3ef87ae543c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"
Jul 31 15:32:59.857797 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.857774    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d091f55f-2add-44b7-9f9f-e3ef87ae543c-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b\" (UID: \"d091f55f-2add-44b7-9f9f-e3ef87ae543c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"
Jul 31 15:32:59.886427 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.886393    2559 generic.go:356] "Generic (PLEG): container finished" podID="d7737771-79a6-4951-a4eb-09b5316f0ee5" containerID="74c5d11f45743fd550d528043552470b861763b85cd0125a2096003081730e41" exitCode=0
Jul 31 15:32:59.886559 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.886451    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq" event={"ID":"d7737771-79a6-4951-a4eb-09b5316f0ee5","Type":"ContainerDied","Data":"74c5d11f45743fd550d528043552470b861763b85cd0125a2096003081730e41"}
Jul 31 15:32:59.886559 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.886476    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"
Jul 31 15:32:59.886559 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.886546    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq" event={"ID":"d7737771-79a6-4951-a4eb-09b5316f0ee5","Type":"ContainerDied","Data":"59ec8abd4649a87d56f85e2dac0097b437a4e9a4f9c9be7b30a2406b7af0fbff"}
Jul 31 15:32:59.886685 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.886586    2559 scope.go:117] "RemoveContainer" containerID="770b1550524740e3e6b2a4c2c01484df1699caedeea41c78d5d3d05898aeb143"
Jul 31 15:32:59.894665 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.894646    2559 scope.go:117] "RemoveContainer" containerID="74c5d11f45743fd550d528043552470b861763b85cd0125a2096003081730e41"
Jul 31 15:32:59.902037 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.902019    2559 scope.go:117] "RemoveContainer" containerID="f5d4bb331773a321074adcd8e2bc2e0d57e328849f4e692581959ee65d0b1fbe"
Jul 31 15:32:59.908623 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.908602    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"]
Jul 31 15:32:59.909264 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.909245    2559 scope.go:117] "RemoveContainer" containerID="770b1550524740e3e6b2a4c2c01484df1699caedeea41c78d5d3d05898aeb143"
Jul 31 15:32:59.909518 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:32:59.909498    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"770b1550524740e3e6b2a4c2c01484df1699caedeea41c78d5d3d05898aeb143\": container with ID starting with 770b1550524740e3e6b2a4c2c01484df1699caedeea41c78d5d3d05898aeb143 not found: ID does not exist" containerID="770b1550524740e3e6b2a4c2c01484df1699caedeea41c78d5d3d05898aeb143"
Jul 31 15:32:59.909591 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.909530    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"770b1550524740e3e6b2a4c2c01484df1699caedeea41c78d5d3d05898aeb143"} err="failed to get container status \"770b1550524740e3e6b2a4c2c01484df1699caedeea41c78d5d3d05898aeb143\": rpc error: code = NotFound desc = could not find container \"770b1550524740e3e6b2a4c2c01484df1699caedeea41c78d5d3d05898aeb143\": container with ID starting with 770b1550524740e3e6b2a4c2c01484df1699caedeea41c78d5d3d05898aeb143 not found: ID does not exist"
Jul 31 15:32:59.909591 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.909557    2559 scope.go:117] "RemoveContainer" containerID="74c5d11f45743fd550d528043552470b861763b85cd0125a2096003081730e41"
Jul 31 15:32:59.909828 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:32:59.909808    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74c5d11f45743fd550d528043552470b861763b85cd0125a2096003081730e41\": container with ID starting with 74c5d11f45743fd550d528043552470b861763b85cd0125a2096003081730e41 not found: ID does not exist" containerID="74c5d11f45743fd550d528043552470b861763b85cd0125a2096003081730e41"
Jul 31 15:32:59.909896 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.909833    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74c5d11f45743fd550d528043552470b861763b85cd0125a2096003081730e41"} err="failed to get container status \"74c5d11f45743fd550d528043552470b861763b85cd0125a2096003081730e41\": rpc error: code = NotFound desc = could not find container \"74c5d11f45743fd550d528043552470b861763b85cd0125a2096003081730e41\": container with ID starting with 74c5d11f45743fd550d528043552470b861763b85cd0125a2096003081730e41 not found: ID does not exist"
Jul 31 15:32:59.909896 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.909850    2559 scope.go:117] "RemoveContainer" containerID="f5d4bb331773a321074adcd8e2bc2e0d57e328849f4e692581959ee65d0b1fbe"
Jul 31 15:32:59.910086 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:32:59.910069    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5d4bb331773a321074adcd8e2bc2e0d57e328849f4e692581959ee65d0b1fbe\": container with ID starting with f5d4bb331773a321074adcd8e2bc2e0d57e328849f4e692581959ee65d0b1fbe not found: ID does not exist" containerID="f5d4bb331773a321074adcd8e2bc2e0d57e328849f4e692581959ee65d0b1fbe"
Jul 31 15:32:59.910126 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.910093    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5d4bb331773a321074adcd8e2bc2e0d57e328849f4e692581959ee65d0b1fbe"} err="failed to get container status \"f5d4bb331773a321074adcd8e2bc2e0d57e328849f4e692581959ee65d0b1fbe\": rpc error: code = NotFound desc = could not find container \"f5d4bb331773a321074adcd8e2bc2e0d57e328849f4e692581959ee65d0b1fbe\": container with ID starting with f5d4bb331773a321074adcd8e2bc2e0d57e328849f4e692581959ee65d0b1fbe not found: ID does not exist"
Jul 31 15:32:59.912951 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:32:59.912932    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-c4rlq"]
Jul 31 15:33:00.120708 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:00.120619    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"
Jul 31 15:33:00.221101 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:00.221066    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7737771-79a6-4951-a4eb-09b5316f0ee5" path="/var/lib/kubelet/pods/d7737771-79a6-4951-a4eb-09b5316f0ee5/volumes"
Jul 31 15:33:00.257621 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:00.257590    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"]
Jul 31 15:33:00.260723 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:33:00.260696    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd091f55f_2add_44b7_9f9f_e3ef87ae543c.slice/crio-53f4d95630793638f5313858f2a6a3e1cbe9a6253b5e1d978fd9fab67c3ae5f2 WatchSource:0}: Error finding container 53f4d95630793638f5313858f2a6a3e1cbe9a6253b5e1d978fd9fab67c3ae5f2: Status 404 returned error can't find the container with id 53f4d95630793638f5313858f2a6a3e1cbe9a6253b5e1d978fd9fab67c3ae5f2
Jul 31 15:33:00.891974 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:00.891929    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b" event={"ID":"d091f55f-2add-44b7-9f9f-e3ef87ae543c","Type":"ContainerStarted","Data":"70a2a5a12066ee566e13cc8d5429ebd228ffa62eec2de4b67bb11d6d40a4affe"}
Jul 31 15:33:00.892188 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:00.892014    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b" event={"ID":"d091f55f-2add-44b7-9f9f-e3ef87ae543c","Type":"ContainerStarted","Data":"53f4d95630793638f5313858f2a6a3e1cbe9a6253b5e1d978fd9fab67c3ae5f2"}
Jul 31 15:33:04.907957 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:04.907924    2559 generic.go:356] "Generic (PLEG): container finished" podID="d091f55f-2add-44b7-9f9f-e3ef87ae543c" containerID="70a2a5a12066ee566e13cc8d5429ebd228ffa62eec2de4b67bb11d6d40a4affe" exitCode=0
Jul 31 15:33:04.908358 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:04.907978    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b" event={"ID":"d091f55f-2add-44b7-9f9f-e3ef87ae543c","Type":"ContainerDied","Data":"70a2a5a12066ee566e13cc8d5429ebd228ffa62eec2de4b67bb11d6d40a4affe"}
Jul 31 15:33:05.913718 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:05.913684    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b" event={"ID":"d091f55f-2add-44b7-9f9f-e3ef87ae543c","Type":"ContainerStarted","Data":"49a19ed7ad412e102050bc4b83f992c670b4c69f7a4ee26faeb6b2df8c752801"}
Jul 31 15:33:05.914104 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:05.913773    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b" event={"ID":"d091f55f-2add-44b7-9f9f-e3ef87ae543c","Type":"ContainerStarted","Data":"343115c3960385156dab467c2ad6f8dcfe77b6f329216f0873d758e4dc270e3d"}
Jul 31 15:33:05.914104 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:05.913967    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"
Jul 31 15:33:05.914104 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:05.914021    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"
Jul 31 15:33:05.937032 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:05.936982    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b" podStartSLOduration=3.054360591 podStartE2EDuration="6.936966796s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.882606205s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:33:05.935583789 +0000 UTC m=+2682.247141468" watchObservedRunningTime="2026-07-31 15:33:05.936966796 +0000 UTC m=+2682.248524418"
Jul 31 15:33:11.924178 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:11.924148    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"
Jul 31 15:33:41.928651 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:41.928618    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"
Jul 31 15:33:49.282821 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:49.282784    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"]
Jul 31 15:33:49.283289 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:49.283014    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b" podUID="d091f55f-2add-44b7-9f9f-e3ef87ae543c" containerName="kserve-container" containerID="cri-o://343115c3960385156dab467c2ad6f8dcfe77b6f329216f0873d758e4dc270e3d" gracePeriod=30
Jul 31 15:33:49.283289 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:49.283070    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b" podUID="d091f55f-2add-44b7-9f9f-e3ef87ae543c" containerName="kube-rbac-proxy" containerID="cri-o://49a19ed7ad412e102050bc4b83f992c670b4c69f7a4ee26faeb6b2df8c752801" gracePeriod=30
Jul 31 15:33:50.077364 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:50.077334    2559 generic.go:356] "Generic (PLEG): container finished" podID="d091f55f-2add-44b7-9f9f-e3ef87ae543c" containerID="49a19ed7ad412e102050bc4b83f992c670b4c69f7a4ee26faeb6b2df8c752801" exitCode=2
Jul 31 15:33:50.077532 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:50.077394    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b" event={"ID":"d091f55f-2add-44b7-9f9f-e3ef87ae543c","Type":"ContainerDied","Data":"49a19ed7ad412e102050bc4b83f992c670b4c69f7a4ee26faeb6b2df8c752801"}
Jul 31 15:33:50.530926 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:50.530901    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"
Jul 31 15:33:50.638317 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:50.638256    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d091f55f-2add-44b7-9f9f-e3ef87ae543c-kserve-provision-location\") pod \"d091f55f-2add-44b7-9f9f-e3ef87ae543c\" (UID: \"d091f55f-2add-44b7-9f9f-e3ef87ae543c\") "
Jul 31 15:33:50.638317 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:50.638288    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d091f55f-2add-44b7-9f9f-e3ef87ae543c-cabundle-cert\") pod \"d091f55f-2add-44b7-9f9f-e3ef87ae543c\" (UID: \"d091f55f-2add-44b7-9f9f-e3ef87ae543c\") "
Jul 31 15:33:50.638317 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:50.638312    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d091f55f-2add-44b7-9f9f-e3ef87ae543c-kube-rbac-proxy-sar-config\") pod \"d091f55f-2add-44b7-9f9f-e3ef87ae543c\" (UID: \"d091f55f-2add-44b7-9f9f-e3ef87ae543c\") "
Jul 31 15:33:50.638534 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:50.638332    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpgc8\" (UniqueName: \"kubernetes.io/projected/d091f55f-2add-44b7-9f9f-e3ef87ae543c-kube-api-access-mpgc8\") pod \"d091f55f-2add-44b7-9f9f-e3ef87ae543c\" (UID: \"d091f55f-2add-44b7-9f9f-e3ef87ae543c\") "
Jul 31 15:33:50.638534 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:50.638370    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d091f55f-2add-44b7-9f9f-e3ef87ae543c-proxy-tls\") pod \"d091f55f-2add-44b7-9f9f-e3ef87ae543c\" (UID: \"d091f55f-2add-44b7-9f9f-e3ef87ae543c\") "
Jul 31 15:33:50.638611 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:50.638586    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d091f55f-2add-44b7-9f9f-e3ef87ae543c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d091f55f-2add-44b7-9f9f-e3ef87ae543c" (UID: "d091f55f-2add-44b7-9f9f-e3ef87ae543c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:33:50.638655 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:50.638616    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d091f55f-2add-44b7-9f9f-e3ef87ae543c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d091f55f-2add-44b7-9f9f-e3ef87ae543c" (UID: "d091f55f-2add-44b7-9f9f-e3ef87ae543c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:33:50.638700 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:50.638673    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d091f55f-2add-44b7-9f9f-e3ef87ae543c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d091f55f-2add-44b7-9f9f-e3ef87ae543c" (UID: "d091f55f-2add-44b7-9f9f-e3ef87ae543c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:33:50.640525 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:50.640504    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d091f55f-2add-44b7-9f9f-e3ef87ae543c-kube-api-access-mpgc8" (OuterVolumeSpecName: "kube-api-access-mpgc8") pod "d091f55f-2add-44b7-9f9f-e3ef87ae543c" (UID: "d091f55f-2add-44b7-9f9f-e3ef87ae543c"). InnerVolumeSpecName "kube-api-access-mpgc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:33:50.640628 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:50.640579    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d091f55f-2add-44b7-9f9f-e3ef87ae543c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d091f55f-2add-44b7-9f9f-e3ef87ae543c" (UID: "d091f55f-2add-44b7-9f9f-e3ef87ae543c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:33:50.739279 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:50.739230    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d091f55f-2add-44b7-9f9f-e3ef87ae543c-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:33:50.739279 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:50.739273    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d091f55f-2add-44b7-9f9f-e3ef87ae543c-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:33:50.739279 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:50.739283    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d091f55f-2add-44b7-9f9f-e3ef87ae543c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:33:50.739279 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:50.739293    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpgc8\" (UniqueName: \"kubernetes.io/projected/d091f55f-2add-44b7-9f9f-e3ef87ae543c-kube-api-access-mpgc8\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:33:50.739560 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:50.739304    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d091f55f-2add-44b7-9f9f-e3ef87ae543c-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:33:51.083073 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:51.082992    2559 generic.go:356] "Generic (PLEG): container finished" podID="d091f55f-2add-44b7-9f9f-e3ef87ae543c" containerID="343115c3960385156dab467c2ad6f8dcfe77b6f329216f0873d758e4dc270e3d" exitCode=0
Jul 31 15:33:51.083073 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:51.083058    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"
Jul 31 15:33:51.083073 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:51.083057    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b" event={"ID":"d091f55f-2add-44b7-9f9f-e3ef87ae543c","Type":"ContainerDied","Data":"343115c3960385156dab467c2ad6f8dcfe77b6f329216f0873d758e4dc270e3d"}
Jul 31 15:33:51.083284 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:51.083133    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b" event={"ID":"d091f55f-2add-44b7-9f9f-e3ef87ae543c","Type":"ContainerDied","Data":"53f4d95630793638f5313858f2a6a3e1cbe9a6253b5e1d978fd9fab67c3ae5f2"}
Jul 31 15:33:51.083284 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:51.083172    2559 scope.go:117] "RemoveContainer" containerID="49a19ed7ad412e102050bc4b83f992c670b4c69f7a4ee26faeb6b2df8c752801"
Jul 31 15:33:51.091555 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:51.091536    2559 scope.go:117] "RemoveContainer" containerID="343115c3960385156dab467c2ad6f8dcfe77b6f329216f0873d758e4dc270e3d"
Jul 31 15:33:51.099061 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:51.099042    2559 scope.go:117] "RemoveContainer" containerID="70a2a5a12066ee566e13cc8d5429ebd228ffa62eec2de4b67bb11d6d40a4affe"
Jul 31 15:33:51.104640 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:51.104618    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"]
Jul 31 15:33:51.106146 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:51.106133    2559 scope.go:117] "RemoveContainer" containerID="49a19ed7ad412e102050bc4b83f992c670b4c69f7a4ee26faeb6b2df8c752801"
Jul 31 15:33:51.106411 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:33:51.106392    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49a19ed7ad412e102050bc4b83f992c670b4c69f7a4ee26faeb6b2df8c752801\": container with ID starting with 49a19ed7ad412e102050bc4b83f992c670b4c69f7a4ee26faeb6b2df8c752801 not found: ID does not exist" containerID="49a19ed7ad412e102050bc4b83f992c670b4c69f7a4ee26faeb6b2df8c752801"
Jul 31 15:33:51.106484 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:51.106424    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49a19ed7ad412e102050bc4b83f992c670b4c69f7a4ee26faeb6b2df8c752801"} err="failed to get container status \"49a19ed7ad412e102050bc4b83f992c670b4c69f7a4ee26faeb6b2df8c752801\": rpc error: code = NotFound desc = could not find container \"49a19ed7ad412e102050bc4b83f992c670b4c69f7a4ee26faeb6b2df8c752801\": container with ID starting with 49a19ed7ad412e102050bc4b83f992c670b4c69f7a4ee26faeb6b2df8c752801 not found: ID does not exist"
Jul 31 15:33:51.106484 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:51.106448    2559 scope.go:117] "RemoveContainer" containerID="343115c3960385156dab467c2ad6f8dcfe77b6f329216f0873d758e4dc270e3d"
Jul 31 15:33:51.106688 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:33:51.106672    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"343115c3960385156dab467c2ad6f8dcfe77b6f329216f0873d758e4dc270e3d\": container with ID starting with 343115c3960385156dab467c2ad6f8dcfe77b6f329216f0873d758e4dc270e3d not found: ID does not exist" containerID="343115c3960385156dab467c2ad6f8dcfe77b6f329216f0873d758e4dc270e3d"
Jul 31 15:33:51.106745 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:51.106693    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"343115c3960385156dab467c2ad6f8dcfe77b6f329216f0873d758e4dc270e3d"} err="failed to get container status \"343115c3960385156dab467c2ad6f8dcfe77b6f329216f0873d758e4dc270e3d\": rpc error: code = NotFound desc = could not find container \"343115c3960385156dab467c2ad6f8dcfe77b6f329216f0873d758e4dc270e3d\": container with ID starting with 343115c3960385156dab467c2ad6f8dcfe77b6f329216f0873d758e4dc270e3d not found: ID does not exist"
Jul 31 15:33:51.106745 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:51.106708    2559 scope.go:117] "RemoveContainer" containerID="70a2a5a12066ee566e13cc8d5429ebd228ffa62eec2de4b67bb11d6d40a4affe"
Jul 31 15:33:51.106920 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:33:51.106906    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70a2a5a12066ee566e13cc8d5429ebd228ffa62eec2de4b67bb11d6d40a4affe\": container with ID starting with 70a2a5a12066ee566e13cc8d5429ebd228ffa62eec2de4b67bb11d6d40a4affe not found: ID does not exist" containerID="70a2a5a12066ee566e13cc8d5429ebd228ffa62eec2de4b67bb11d6d40a4affe"
Jul 31 15:33:51.106961 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:51.106923    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70a2a5a12066ee566e13cc8d5429ebd228ffa62eec2de4b67bb11d6d40a4affe"} err="failed to get container status \"70a2a5a12066ee566e13cc8d5429ebd228ffa62eec2de4b67bb11d6d40a4affe\": rpc error: code = NotFound desc = could not find container \"70a2a5a12066ee566e13cc8d5429ebd228ffa62eec2de4b67bb11d6d40a4affe\": container with ID starting with 70a2a5a12066ee566e13cc8d5429ebd228ffa62eec2de4b67bb11d6d40a4affe not found: ID does not exist"
Jul 31 15:33:51.110617 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:51.110597    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-sqw8b"]
Jul 31 15:33:52.220594 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:52.220557    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d091f55f-2add-44b7-9f9f-e3ef87ae543c" path="/var/lib/kubelet/pods/d091f55f-2add-44b7-9f9f-e3ef87ae543c/volumes"
Jul 31 15:33:54.457125 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.457089    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"]
Jul 31 15:33:54.457508 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.457346    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d091f55f-2add-44b7-9f9f-e3ef87ae543c" containerName="kserve-container"
Jul 31 15:33:54.457508 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.457356    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d091f55f-2add-44b7-9f9f-e3ef87ae543c" containerName="kserve-container"
Jul 31 15:33:54.457508 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.457365    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d091f55f-2add-44b7-9f9f-e3ef87ae543c" containerName="storage-initializer"
Jul 31 15:33:54.457508 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.457370    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d091f55f-2add-44b7-9f9f-e3ef87ae543c" containerName="storage-initializer"
Jul 31 15:33:54.457508 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.457388    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d091f55f-2add-44b7-9f9f-e3ef87ae543c" containerName="kube-rbac-proxy"
Jul 31 15:33:54.457508 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.457393    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d091f55f-2add-44b7-9f9f-e3ef87ae543c" containerName="kube-rbac-proxy"
Jul 31 15:33:54.457508 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.457439    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d091f55f-2add-44b7-9f9f-e3ef87ae543c" containerName="kube-rbac-proxy"
Jul 31 15:33:54.457508 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.457449    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d091f55f-2add-44b7-9f9f-e3ef87ae543c" containerName="kserve-container"
Jul 31 15:33:54.462217 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.462195    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"
Jul 31 15:33:54.464671 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.464647    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 15:33:54.464824 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.464706    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Jul 31 15:33:54.464824 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.464759    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Jul 31 15:33:54.464961 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.464761    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 15:33:54.464961 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.464842    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 15:33:54.465247 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.465220    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dd67w\""
Jul 31 15:33:54.465353 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.465220    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:33:54.471495 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.471453    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"]
Jul 31 15:33:54.565445 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.565411    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40c2992b-3bbe-4842-a921-d3909b6e15c5-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-557779745-n8bpb\" (UID: \"40c2992b-3bbe-4842-a921-d3909b6e15c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"
Jul 31 15:33:54.565445 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.565447    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40c2992b-3bbe-4842-a921-d3909b6e15c5-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-557779745-n8bpb\" (UID: \"40c2992b-3bbe-4842-a921-d3909b6e15c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"
Jul 31 15:33:54.565661 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.565467    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40c2992b-3bbe-4842-a921-d3909b6e15c5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-557779745-n8bpb\" (UID: \"40c2992b-3bbe-4842-a921-d3909b6e15c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"
Jul 31 15:33:54.565661 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.565542    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trjqq\" (UniqueName: \"kubernetes.io/projected/40c2992b-3bbe-4842-a921-d3909b6e15c5-kube-api-access-trjqq\") pod \"isvc-sklearn-mcp-predictor-557779745-n8bpb\" (UID: \"40c2992b-3bbe-4842-a921-d3909b6e15c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"
Jul 31 15:33:54.565661 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.565582    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40c2992b-3bbe-4842-a921-d3909b6e15c5-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-557779745-n8bpb\" (UID: \"40c2992b-3bbe-4842-a921-d3909b6e15c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"
Jul 31 15:33:54.666351 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.666297    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40c2992b-3bbe-4842-a921-d3909b6e15c5-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-557779745-n8bpb\" (UID: \"40c2992b-3bbe-4842-a921-d3909b6e15c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"
Jul 31 15:33:54.666351 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.666352    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40c2992b-3bbe-4842-a921-d3909b6e15c5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-557779745-n8bpb\" (UID: \"40c2992b-3bbe-4842-a921-d3909b6e15c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"
Jul 31 15:33:54.666590 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.666396    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trjqq\" (UniqueName: \"kubernetes.io/projected/40c2992b-3bbe-4842-a921-d3909b6e15c5-kube-api-access-trjqq\") pod \"isvc-sklearn-mcp-predictor-557779745-n8bpb\" (UID: \"40c2992b-3bbe-4842-a921-d3909b6e15c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"
Jul 31 15:33:54.666590 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.666415    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40c2992b-3bbe-4842-a921-d3909b6e15c5-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-557779745-n8bpb\" (UID: \"40c2992b-3bbe-4842-a921-d3909b6e15c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"
Jul 31 15:33:54.666590 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:33:54.666460    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-serving-cert: secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Jul 31 15:33:54.666590 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:33:54.666533    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40c2992b-3bbe-4842-a921-d3909b6e15c5-proxy-tls podName:40c2992b-3bbe-4842-a921-d3909b6e15c5 nodeName:}" failed. No retries permitted until 2026-07-31 15:33:55.166515617 +0000 UTC m=+2731.478073219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/40c2992b-3bbe-4842-a921-d3909b6e15c5-proxy-tls") pod "isvc-sklearn-mcp-predictor-557779745-n8bpb" (UID: "40c2992b-3bbe-4842-a921-d3909b6e15c5") : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Jul 31 15:33:54.666590 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.666570    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40c2992b-3bbe-4842-a921-d3909b6e15c5-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-557779745-n8bpb\" (UID: \"40c2992b-3bbe-4842-a921-d3909b6e15c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"
Jul 31 15:33:54.666879 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.666859    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40c2992b-3bbe-4842-a921-d3909b6e15c5-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-557779745-n8bpb\" (UID: \"40c2992b-3bbe-4842-a921-d3909b6e15c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"
Jul 31 15:33:54.667118 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.667101    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40c2992b-3bbe-4842-a921-d3909b6e15c5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-557779745-n8bpb\" (UID: \"40c2992b-3bbe-4842-a921-d3909b6e15c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"
Jul 31 15:33:54.667155 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.667135    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40c2992b-3bbe-4842-a921-d3909b6e15c5-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-557779745-n8bpb\" (UID: \"40c2992b-3bbe-4842-a921-d3909b6e15c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"
Jul 31 15:33:54.677380 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:54.677348    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trjqq\" (UniqueName: \"kubernetes.io/projected/40c2992b-3bbe-4842-a921-d3909b6e15c5-kube-api-access-trjqq\") pod \"isvc-sklearn-mcp-predictor-557779745-n8bpb\" (UID: \"40c2992b-3bbe-4842-a921-d3909b6e15c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"
Jul 31 15:33:55.171345 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:55.171285    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40c2992b-3bbe-4842-a921-d3909b6e15c5-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-557779745-n8bpb\" (UID: \"40c2992b-3bbe-4842-a921-d3909b6e15c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"
Jul 31 15:33:55.173867 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:55.173844    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40c2992b-3bbe-4842-a921-d3909b6e15c5-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-557779745-n8bpb\" (UID: \"40c2992b-3bbe-4842-a921-d3909b6e15c5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"
Jul 31 15:33:55.373796 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:55.373758    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"
Jul 31 15:33:55.510538 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:55.510515    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"]
Jul 31 15:33:55.513114 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:33:55.513076    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c2992b_3bbe_4842_a921_d3909b6e15c5.slice/crio-8e900a3c83cc32c1ae4206ca1da8b1e8e33b04e318a77a263ae359763254b1f8 WatchSource:0}: Error finding container 8e900a3c83cc32c1ae4206ca1da8b1e8e33b04e318a77a263ae359763254b1f8: Status 404 returned error can't find the container with id 8e900a3c83cc32c1ae4206ca1da8b1e8e33b04e318a77a263ae359763254b1f8
Jul 31 15:33:56.107101 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:56.107066    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb" event={"ID":"40c2992b-3bbe-4842-a921-d3909b6e15c5","Type":"ContainerStarted","Data":"a72cc8459dfc585612063637da9ba1a2448d831fecc9cbfa97eed7d13525f513"}
Jul 31 15:33:56.107287 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:33:56.107129    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb" event={"ID":"40c2992b-3bbe-4842-a921-d3909b6e15c5","Type":"ContainerStarted","Data":"8e900a3c83cc32c1ae4206ca1da8b1e8e33b04e318a77a263ae359763254b1f8"}
Jul 31 15:34:00.122802 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:00.122750    2559 generic.go:356] "Generic (PLEG): container finished" podID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerID="a72cc8459dfc585612063637da9ba1a2448d831fecc9cbfa97eed7d13525f513" exitCode=0
Jul 31 15:34:00.122802 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:00.122813    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb" event={"ID":"40c2992b-3bbe-4842-a921-d3909b6e15c5","Type":"ContainerDied","Data":"a72cc8459dfc585612063637da9ba1a2448d831fecc9cbfa97eed7d13525f513"}
Jul 31 15:34:00.182692 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:00.182674    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 15:34:01.131303 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:01.131256    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb" event={"ID":"40c2992b-3bbe-4842-a921-d3909b6e15c5","Type":"ContainerStarted","Data":"d37a87e353717ae89a65c70eb1e25709a9de7c20a5b7653c94b07aeb939369f2"}
Jul 31 15:34:03.141263 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:03.141230    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb" event={"ID":"40c2992b-3bbe-4842-a921-d3909b6e15c5","Type":"ContainerStarted","Data":"28fd0600c22a5a54bef04c0e559419b5b39b7f73dd8faab76211da6bd5e0a867"}
Jul 31 15:34:03.141650 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:03.141291    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb" event={"ID":"40c2992b-3bbe-4842-a921-d3909b6e15c5","Type":"ContainerStarted","Data":"b7577a176dd4c423c371191c45ed57672648d88cb2288bd13a6e05996c9a35d8"}
Jul 31 15:34:03.141650 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:03.141398    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"
Jul 31 15:34:03.141650 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:03.141463    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"
Jul 31 15:34:03.165202 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:03.165139    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb" podStartSLOduration=3.194958658 podStartE2EDuration="9.165122625s" totalImagesPullingTime="2.416764063s" totalInitContainerRuntime="3.553399904s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:34:03.163175528 +0000 UTC m=+2739.474733150" watchObservedRunningTime="2026-07-31 15:34:03.165122625 +0000 UTC m=+2739.476680247"
Jul 31 15:34:04.145239 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:04.145207    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"
Jul 31 15:34:10.153448 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:10.153419    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"
Jul 31 15:34:30.154285 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:30.154240    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb" podUID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.55:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 31 15:34:35.374327 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:35.374284    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb" podUID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.55:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 31 15:34:40.158651 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:40.158621    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"
Jul 31 15:34:42.041513 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:42.041475    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd"]
Jul 31 15:34:42.044061 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:42.044042    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd"
Jul 31 15:34:42.046558 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:42.046535    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Jul 31 15:34:42.046671 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:42.046555    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Jul 31 15:34:42.046671 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:42.046569    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 31 15:34:42.054771 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:42.054747    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd"]
Jul 31 15:34:42.130455 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:42.130410    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e44cf494-c6bd-4597-a84d-f6cb49f591e2-proxy-tls\") pod \"isvc-sklearn-s3-predictor-567fcf85f4-snpmd\" (UID: \"e44cf494-c6bd-4597-a84d-f6cb49f591e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd"
Jul 31 15:34:42.130455 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:42.130455    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e44cf494-c6bd-4597-a84d-f6cb49f591e2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-567fcf85f4-snpmd\" (UID: \"e44cf494-c6bd-4597-a84d-f6cb49f591e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd"
Jul 31 15:34:42.130689 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:42.130510    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e44cf494-c6bd-4597-a84d-f6cb49f591e2-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-567fcf85f4-snpmd\" (UID: \"e44cf494-c6bd-4597-a84d-f6cb49f591e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd"
Jul 31 15:34:42.130689 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:42.130553    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nfp9\" (UniqueName: \"kubernetes.io/projected/e44cf494-c6bd-4597-a84d-f6cb49f591e2-kube-api-access-9nfp9\") pod \"isvc-sklearn-s3-predictor-567fcf85f4-snpmd\" (UID: \"e44cf494-c6bd-4597-a84d-f6cb49f591e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd"
Jul 31 15:34:42.231792 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:42.231760    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e44cf494-c6bd-4597-a84d-f6cb49f591e2-proxy-tls\") pod \"isvc-sklearn-s3-predictor-567fcf85f4-snpmd\" (UID: \"e44cf494-c6bd-4597-a84d-f6cb49f591e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd"
Jul 31 15:34:42.231987 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:42.231798    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e44cf494-c6bd-4597-a84d-f6cb49f591e2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-567fcf85f4-snpmd\" (UID: \"e44cf494-c6bd-4597-a84d-f6cb49f591e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd"
Jul 31 15:34:42.231987 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:42.231848    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e44cf494-c6bd-4597-a84d-f6cb49f591e2-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-567fcf85f4-snpmd\" (UID: \"e44cf494-c6bd-4597-a84d-f6cb49f591e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd"
Jul 31 15:34:42.231987 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:42.231881    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nfp9\" (UniqueName: \"kubernetes.io/projected/e44cf494-c6bd-4597-a84d-f6cb49f591e2-kube-api-access-9nfp9\") pod \"isvc-sklearn-s3-predictor-567fcf85f4-snpmd\" (UID: \"e44cf494-c6bd-4597-a84d-f6cb49f591e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd"
Jul 31 15:34:42.232405 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:42.232382    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e44cf494-c6bd-4597-a84d-f6cb49f591e2-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-567fcf85f4-snpmd\" (UID: \"e44cf494-c6bd-4597-a84d-f6cb49f591e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd"
Jul 31 15:34:42.232689 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:42.232635    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e44cf494-c6bd-4597-a84d-f6cb49f591e2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-567fcf85f4-snpmd\" (UID: \"e44cf494-c6bd-4597-a84d-f6cb49f591e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd"
Jul 31 15:34:42.234384 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:42.234365    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e44cf494-c6bd-4597-a84d-f6cb49f591e2-proxy-tls\") pod \"isvc-sklearn-s3-predictor-567fcf85f4-snpmd\" (UID: \"e44cf494-c6bd-4597-a84d-f6cb49f591e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd"
Jul 31 15:34:42.240152 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:42.240125    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nfp9\" (UniqueName: \"kubernetes.io/projected/e44cf494-c6bd-4597-a84d-f6cb49f591e2-kube-api-access-9nfp9\") pod \"isvc-sklearn-s3-predictor-567fcf85f4-snpmd\" (UID: \"e44cf494-c6bd-4597-a84d-f6cb49f591e2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd"
Jul 31 15:34:42.355323 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:42.355290    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd"
Jul 31 15:34:42.483888 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:42.483862    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd"]
Jul 31 15:34:42.486484 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:34:42.486455    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode44cf494_c6bd_4597_a84d_f6cb49f591e2.slice/crio-5ea29306fa5d4fc8a9ca0e1e904c87419fab75e4ad882413dc84ab5271038211 WatchSource:0}: Error finding container 5ea29306fa5d4fc8a9ca0e1e904c87419fab75e4ad882413dc84ab5271038211: Status 404 returned error can't find the container with id 5ea29306fa5d4fc8a9ca0e1e904c87419fab75e4ad882413dc84ab5271038211
Jul 31 15:34:43.288182 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:43.288146    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd" event={"ID":"e44cf494-c6bd-4597-a84d-f6cb49f591e2","Type":"ContainerStarted","Data":"72d18536008a74f6755f261ea9ae632cb5de6736080649b838fc6570719bea34"}
Jul 31 15:34:43.288593 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:43.288208    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd" event={"ID":"e44cf494-c6bd-4597-a84d-f6cb49f591e2","Type":"ContainerStarted","Data":"5ea29306fa5d4fc8a9ca0e1e904c87419fab75e4ad882413dc84ab5271038211"}
Jul 31 15:34:44.293327 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:44.293291    2559 generic.go:356] "Generic (PLEG): container finished" podID="e44cf494-c6bd-4597-a84d-f6cb49f591e2" containerID="72d18536008a74f6755f261ea9ae632cb5de6736080649b838fc6570719bea34" exitCode=0
Jul 31 15:34:44.293716 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:44.293348    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd" event={"ID":"e44cf494-c6bd-4597-a84d-f6cb49f591e2","Type":"ContainerDied","Data":"72d18536008a74f6755f261ea9ae632cb5de6736080649b838fc6570719bea34"}
Jul 31 15:34:45.298849 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:45.298758    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd" event={"ID":"e44cf494-c6bd-4597-a84d-f6cb49f591e2","Type":"ContainerStarted","Data":"522cfa8f02af711319d8e938a997921676a74c1a1e41d043db0585a424a7eefa"}
Jul 31 15:34:45.298849 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:45.298817    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd" event={"ID":"e44cf494-c6bd-4597-a84d-f6cb49f591e2","Type":"ContainerStarted","Data":"97825ba273e59ec4e035d32bf499e4c1caca83c502c35d4f6de9b04eff459dd1"}
Jul 31 15:34:45.299375 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:45.298903    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd"
Jul 31 15:34:45.318606 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:45.318551    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd" podStartSLOduration=2.3010669679999998 podStartE2EDuration="3.31853152s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.017464552s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:34:45.318041622 +0000 UTC m=+2781.629599243" watchObservedRunningTime="2026-07-31 15:34:45.31853152 +0000 UTC m=+2781.630089143"
Jul 31 15:34:46.303243 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:46.303207    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd"
Jul 31 15:34:46.304494 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:46.304463    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd" podUID="e44cf494-c6bd-4597-a84d-f6cb49f591e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 31 15:34:47.307407 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:47.307368    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd" podUID="e44cf494-c6bd-4597-a84d-f6cb49f591e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 31 15:34:52.312109 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:52.312079    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd"
Jul 31 15:34:52.312555 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:34:52.312531    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd" podUID="e44cf494-c6bd-4597-a84d-f6cb49f591e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 31 15:35:02.313071 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:02.313028    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd" podUID="e44cf494-c6bd-4597-a84d-f6cb49f591e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 31 15:35:10.160094 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:10.160059    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"
Jul 31 15:35:12.313195 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:12.313153    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd" podUID="e44cf494-c6bd-4597-a84d-f6cb49f591e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 31 15:35:14.549289 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:14.549247    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"]
Jul 31 15:35:14.549654 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:14.549543    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb" podUID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerName="kserve-container" containerID="cri-o://d37a87e353717ae89a65c70eb1e25709a9de7c20a5b7653c94b07aeb939369f2" gracePeriod=30
Jul 31 15:35:14.549765 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:14.549634    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb" podUID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerName="kube-rbac-proxy" containerID="cri-o://28fd0600c22a5a54bef04c0e559419b5b39b7f73dd8faab76211da6bd5e0a867" gracePeriod=30
Jul 31 15:35:14.549765 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:14.549663    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb" podUID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerName="kserve-agent" containerID="cri-o://b7577a176dd4c423c371191c45ed57672648d88cb2288bd13a6e05996c9a35d8" gracePeriod=30
Jul 31 15:35:15.149755 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:15.149693    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb" podUID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Jul 31 15:35:15.418527 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:15.418446    2559 generic.go:356] "Generic (PLEG): container finished" podID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerID="28fd0600c22a5a54bef04c0e559419b5b39b7f73dd8faab76211da6bd5e0a867" exitCode=2
Jul 31 15:35:15.418527 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:15.418515    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb" event={"ID":"40c2992b-3bbe-4842-a921-d3909b6e15c5","Type":"ContainerDied","Data":"28fd0600c22a5a54bef04c0e559419b5b39b7f73dd8faab76211da6bd5e0a867"}
Jul 31 15:35:17.428689 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:17.428660    2559 generic.go:356] "Generic (PLEG): container finished" podID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerID="d37a87e353717ae89a65c70eb1e25709a9de7c20a5b7653c94b07aeb939369f2" exitCode=0
Jul 31 15:35:17.429110 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:17.428724    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb" event={"ID":"40c2992b-3bbe-4842-a921-d3909b6e15c5","Type":"ContainerDied","Data":"d37a87e353717ae89a65c70eb1e25709a9de7c20a5b7653c94b07aeb939369f2"}
Jul 31 15:35:20.149109 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:20.149061    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb" podUID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Jul 31 15:35:20.154514 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:20.154484    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb" podUID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.55:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 31 15:35:22.312666 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:22.312618    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd" podUID="e44cf494-c6bd-4597-a84d-f6cb49f591e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 31 15:35:25.149691 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:25.149644    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb" podUID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Jul 31 15:35:25.150147 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:25.149814    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"
Jul 31 15:35:30.149281 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:30.149235    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb" podUID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Jul 31 15:35:30.154604 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:30.154576    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb" podUID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.55:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 31 15:35:32.312521 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:32.312481    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd" podUID="e44cf494-c6bd-4597-a84d-f6cb49f591e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 31 15:35:35.149685 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:35.149641    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb" podUID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Jul 31 15:35:40.149573 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:40.149529    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb" podUID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Jul 31 15:35:40.155010 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:40.154984    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb" podUID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.55:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 31 15:35:40.155138 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:40.155126    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"
Jul 31 15:35:42.313261 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:42.313226    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd" podUID="e44cf494-c6bd-4597-a84d-f6cb49f591e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 31 15:35:44.725704 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:44.725678    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"
Jul 31 15:35:44.812146 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:44.812056    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40c2992b-3bbe-4842-a921-d3909b6e15c5-proxy-tls\") pod \"40c2992b-3bbe-4842-a921-d3909b6e15c5\" (UID: \"40c2992b-3bbe-4842-a921-d3909b6e15c5\") "
Jul 31 15:35:44.812146 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:44.812098    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trjqq\" (UniqueName: \"kubernetes.io/projected/40c2992b-3bbe-4842-a921-d3909b6e15c5-kube-api-access-trjqq\") pod \"40c2992b-3bbe-4842-a921-d3909b6e15c5\" (UID: \"40c2992b-3bbe-4842-a921-d3909b6e15c5\") "
Jul 31 15:35:44.812146 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:44.812139    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40c2992b-3bbe-4842-a921-d3909b6e15c5-cabundle-cert\") pod \"40c2992b-3bbe-4842-a921-d3909b6e15c5\" (UID: \"40c2992b-3bbe-4842-a921-d3909b6e15c5\") "
Jul 31 15:35:44.812382 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:44.812164    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40c2992b-3bbe-4842-a921-d3909b6e15c5-kserve-provision-location\") pod \"40c2992b-3bbe-4842-a921-d3909b6e15c5\" (UID: \"40c2992b-3bbe-4842-a921-d3909b6e15c5\") "
Jul 31 15:35:44.812382 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:44.812201    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40c2992b-3bbe-4842-a921-d3909b6e15c5-kube-rbac-proxy-sar-config\") pod \"40c2992b-3bbe-4842-a921-d3909b6e15c5\" (UID: \"40c2992b-3bbe-4842-a921-d3909b6e15c5\") "
Jul 31 15:35:44.812552 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:44.812522    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40c2992b-3bbe-4842-a921-d3909b6e15c5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "40c2992b-3bbe-4842-a921-d3909b6e15c5" (UID: "40c2992b-3bbe-4842-a921-d3909b6e15c5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:35:44.812614 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:44.812564    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c2992b-3bbe-4842-a921-d3909b6e15c5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "40c2992b-3bbe-4842-a921-d3909b6e15c5" (UID: "40c2992b-3bbe-4842-a921-d3909b6e15c5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:35:44.812614 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:44.812577    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c2992b-3bbe-4842-a921-d3909b6e15c5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "40c2992b-3bbe-4842-a921-d3909b6e15c5" (UID: "40c2992b-3bbe-4842-a921-d3909b6e15c5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:35:44.814420 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:44.814396    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c2992b-3bbe-4842-a921-d3909b6e15c5-kube-api-access-trjqq" (OuterVolumeSpecName: "kube-api-access-trjqq") pod "40c2992b-3bbe-4842-a921-d3909b6e15c5" (UID: "40c2992b-3bbe-4842-a921-d3909b6e15c5"). InnerVolumeSpecName "kube-api-access-trjqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:35:44.814420 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:44.814410    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40c2992b-3bbe-4842-a921-d3909b6e15c5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "40c2992b-3bbe-4842-a921-d3909b6e15c5" (UID: "40c2992b-3bbe-4842-a921-d3909b6e15c5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:35:44.912785 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:44.912751    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40c2992b-3bbe-4842-a921-d3909b6e15c5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:35:44.912785 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:44.912779    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40c2992b-3bbe-4842-a921-d3909b6e15c5-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:35:44.912785 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:44.912790    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trjqq\" (UniqueName: \"kubernetes.io/projected/40c2992b-3bbe-4842-a921-d3909b6e15c5-kube-api-access-trjqq\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:35:44.912785 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:44.912799    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40c2992b-3bbe-4842-a921-d3909b6e15c5-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:35:44.913061 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:44.912808    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40c2992b-3bbe-4842-a921-d3909b6e15c5-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:35:45.538175 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:45.538142    2559 generic.go:356] "Generic (PLEG): container finished" podID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerID="b7577a176dd4c423c371191c45ed57672648d88cb2288bd13a6e05996c9a35d8" exitCode=137
Jul 31 15:35:45.538374 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:45.538229    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"
Jul 31 15:35:45.538374 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:45.538229    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb" event={"ID":"40c2992b-3bbe-4842-a921-d3909b6e15c5","Type":"ContainerDied","Data":"b7577a176dd4c423c371191c45ed57672648d88cb2288bd13a6e05996c9a35d8"}
Jul 31 15:35:45.538374 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:45.538295    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb" event={"ID":"40c2992b-3bbe-4842-a921-d3909b6e15c5","Type":"ContainerDied","Data":"8e900a3c83cc32c1ae4206ca1da8b1e8e33b04e318a77a263ae359763254b1f8"}
Jul 31 15:35:45.538374 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:45.538325    2559 scope.go:117] "RemoveContainer" containerID="28fd0600c22a5a54bef04c0e559419b5b39b7f73dd8faab76211da6bd5e0a867"
Jul 31 15:35:45.547078 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:45.547058    2559 scope.go:117] "RemoveContainer" containerID="b7577a176dd4c423c371191c45ed57672648d88cb2288bd13a6e05996c9a35d8"
Jul 31 15:35:45.554558 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:45.554361    2559 scope.go:117] "RemoveContainer" containerID="d37a87e353717ae89a65c70eb1e25709a9de7c20a5b7653c94b07aeb939369f2"
Jul 31 15:35:45.561356 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:45.561332    2559 scope.go:117] "RemoveContainer" containerID="a72cc8459dfc585612063637da9ba1a2448d831fecc9cbfa97eed7d13525f513"
Jul 31 15:35:45.563719 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:45.563698    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"]
Jul 31 15:35:45.568038 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:45.568009    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-557779745-n8bpb"]
Jul 31 15:35:45.570304 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:45.570288    2559 scope.go:117] "RemoveContainer" containerID="28fd0600c22a5a54bef04c0e559419b5b39b7f73dd8faab76211da6bd5e0a867"
Jul 31 15:35:45.570578 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:35:45.570548    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28fd0600c22a5a54bef04c0e559419b5b39b7f73dd8faab76211da6bd5e0a867\": container with ID starting with 28fd0600c22a5a54bef04c0e559419b5b39b7f73dd8faab76211da6bd5e0a867 not found: ID does not exist" containerID="28fd0600c22a5a54bef04c0e559419b5b39b7f73dd8faab76211da6bd5e0a867"
Jul 31 15:35:45.570716 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:45.570571    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28fd0600c22a5a54bef04c0e559419b5b39b7f73dd8faab76211da6bd5e0a867"} err="failed to get container status \"28fd0600c22a5a54bef04c0e559419b5b39b7f73dd8faab76211da6bd5e0a867\": rpc error: code = NotFound desc = could not find container \"28fd0600c22a5a54bef04c0e559419b5b39b7f73dd8faab76211da6bd5e0a867\": container with ID starting with 28fd0600c22a5a54bef04c0e559419b5b39b7f73dd8faab76211da6bd5e0a867 not found: ID does not exist"
Jul 31 15:35:45.570716 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:45.570590    2559 scope.go:117] "RemoveContainer" containerID="b7577a176dd4c423c371191c45ed57672648d88cb2288bd13a6e05996c9a35d8"
Jul 31 15:35:45.570875 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:35:45.570817    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7577a176dd4c423c371191c45ed57672648d88cb2288bd13a6e05996c9a35d8\": container with ID starting with b7577a176dd4c423c371191c45ed57672648d88cb2288bd13a6e05996c9a35d8 not found: ID does not exist" containerID="b7577a176dd4c423c371191c45ed57672648d88cb2288bd13a6e05996c9a35d8"
Jul 31 15:35:45.570875 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:45.570839    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7577a176dd4c423c371191c45ed57672648d88cb2288bd13a6e05996c9a35d8"} err="failed to get container status \"b7577a176dd4c423c371191c45ed57672648d88cb2288bd13a6e05996c9a35d8\": rpc error: code = NotFound desc = could not find container \"b7577a176dd4c423c371191c45ed57672648d88cb2288bd13a6e05996c9a35d8\": container with ID starting with b7577a176dd4c423c371191c45ed57672648d88cb2288bd13a6e05996c9a35d8 not found: ID does not exist"
Jul 31 15:35:45.570875 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:45.570856    2559 scope.go:117] "RemoveContainer" containerID="d37a87e353717ae89a65c70eb1e25709a9de7c20a5b7653c94b07aeb939369f2"
Jul 31 15:35:45.571100 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:35:45.571081    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37a87e353717ae89a65c70eb1e25709a9de7c20a5b7653c94b07aeb939369f2\": container with ID starting with d37a87e353717ae89a65c70eb1e25709a9de7c20a5b7653c94b07aeb939369f2 not found: ID does not exist" containerID="d37a87e353717ae89a65c70eb1e25709a9de7c20a5b7653c94b07aeb939369f2"
Jul 31 15:35:45.571161 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:45.571106    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37a87e353717ae89a65c70eb1e25709a9de7c20a5b7653c94b07aeb939369f2"} err="failed to get container status \"d37a87e353717ae89a65c70eb1e25709a9de7c20a5b7653c94b07aeb939369f2\": rpc error: code = NotFound desc = could not find container \"d37a87e353717ae89a65c70eb1e25709a9de7c20a5b7653c94b07aeb939369f2\": container with ID starting with d37a87e353717ae89a65c70eb1e25709a9de7c20a5b7653c94b07aeb939369f2 not found: ID does not exist"
Jul 31 15:35:45.571161 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:45.571121    2559 scope.go:117] "RemoveContainer" containerID="a72cc8459dfc585612063637da9ba1a2448d831fecc9cbfa97eed7d13525f513"
Jul 31 15:35:45.571356 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:35:45.571338    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a72cc8459dfc585612063637da9ba1a2448d831fecc9cbfa97eed7d13525f513\": container with ID starting with a72cc8459dfc585612063637da9ba1a2448d831fecc9cbfa97eed7d13525f513 not found: ID does not exist" containerID="a72cc8459dfc585612063637da9ba1a2448d831fecc9cbfa97eed7d13525f513"
Jul 31 15:35:45.571453 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:45.571357    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a72cc8459dfc585612063637da9ba1a2448d831fecc9cbfa97eed7d13525f513"} err="failed to get container status \"a72cc8459dfc585612063637da9ba1a2448d831fecc9cbfa97eed7d13525f513\": rpc error: code = NotFound desc = could not find container \"a72cc8459dfc585612063637da9ba1a2448d831fecc9cbfa97eed7d13525f513\": container with ID starting with a72cc8459dfc585612063637da9ba1a2448d831fecc9cbfa97eed7d13525f513 not found: ID does not exist"
Jul 31 15:35:46.221115 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:46.221065    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40c2992b-3bbe-4842-a921-d3909b6e15c5" path="/var/lib/kubelet/pods/40c2992b-3bbe-4842-a921-d3909b6e15c5/volumes"
Jul 31 15:35:52.312993 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:35:52.312948    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd" podUID="e44cf494-c6bd-4597-a84d-f6cb49f591e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 31 15:36:02.220200 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:02.220168    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd"
Jul 31 15:36:12.145719 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.145682    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd"]
Jul 31 15:36:12.146140 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.145930    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd" podUID="e44cf494-c6bd-4597-a84d-f6cb49f591e2" containerName="kserve-container" containerID="cri-o://97825ba273e59ec4e035d32bf499e4c1caca83c502c35d4f6de9b04eff459dd1" gracePeriod=30
Jul 31 15:36:12.146140 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.145981    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd" podUID="e44cf494-c6bd-4597-a84d-f6cb49f591e2" containerName="kube-rbac-proxy" containerID="cri-o://522cfa8f02af711319d8e938a997921676a74c1a1e41d043db0585a424a7eefa" gracePeriod=30
Jul 31 15:36:12.216962 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.216915    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd" podUID="e44cf494-c6bd-4597-a84d-f6cb49f591e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 31 15:36:12.307674 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.307628    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd" podUID="e44cf494-c6bd-4597-a84d-f6cb49f591e2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Jul 31 15:36:12.310786 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.310752    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"]
Jul 31 15:36:12.311033 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.311021    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerName="kserve-container"
Jul 31 15:36:12.311088 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.311035    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerName="kserve-container"
Jul 31 15:36:12.311088 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.311051    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerName="storage-initializer"
Jul 31 15:36:12.311088 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.311056    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerName="storage-initializer"
Jul 31 15:36:12.311088 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.311070    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerName="kube-rbac-proxy"
Jul 31 15:36:12.311088 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.311075    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerName="kube-rbac-proxy"
Jul 31 15:36:12.311088 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.311083    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerName="kserve-agent"
Jul 31 15:36:12.311088 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.311088    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerName="kserve-agent"
Jul 31 15:36:12.311332 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.311138    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerName="kserve-agent"
Jul 31 15:36:12.311332 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.311146    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerName="kserve-container"
Jul 31 15:36:12.311332 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.311153    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="40c2992b-3bbe-4842-a921-d3909b6e15c5" containerName="kube-rbac-proxy"
Jul 31 15:36:12.314049 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.314032    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"
Jul 31 15:36:12.316417 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.316385    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Jul 31 15:36:12.316417 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.316392    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Jul 31 15:36:12.316595 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.316471    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:36:12.326167 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.326139    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"]
Jul 31 15:36:12.414885 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.414791    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bceb285-968e-4147-b902-6c85c9d11fd0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2\" (UID: \"5bceb285-968e-4147-b902-6c85c9d11fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"
Jul 31 15:36:12.414885 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.414842    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bceb285-968e-4147-b902-6c85c9d11fd0-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2\" (UID: \"5bceb285-968e-4147-b902-6c85c9d11fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"
Jul 31 15:36:12.415088 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.414903    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnp25\" (UniqueName: \"kubernetes.io/projected/5bceb285-968e-4147-b902-6c85c9d11fd0-kube-api-access-fnp25\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2\" (UID: \"5bceb285-968e-4147-b902-6c85c9d11fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"
Jul 31 15:36:12.415088 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.414936    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bceb285-968e-4147-b902-6c85c9d11fd0-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2\" (UID: \"5bceb285-968e-4147-b902-6c85c9d11fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"
Jul 31 15:36:12.415088 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.414984    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bceb285-968e-4147-b902-6c85c9d11fd0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2\" (UID: \"5bceb285-968e-4147-b902-6c85c9d11fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"
Jul 31 15:36:12.515982 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.515938    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bceb285-968e-4147-b902-6c85c9d11fd0-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2\" (UID: \"5bceb285-968e-4147-b902-6c85c9d11fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"
Jul 31 15:36:12.516152 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.516018    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bceb285-968e-4147-b902-6c85c9d11fd0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2\" (UID: \"5bceb285-968e-4147-b902-6c85c9d11fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"
Jul 31 15:36:12.516152 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.516119    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bceb285-968e-4147-b902-6c85c9d11fd0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2\" (UID: \"5bceb285-968e-4147-b902-6c85c9d11fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"
Jul 31 15:36:12.516256 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.516171    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bceb285-968e-4147-b902-6c85c9d11fd0-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2\" (UID: \"5bceb285-968e-4147-b902-6c85c9d11fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"
Jul 31 15:36:12.516256 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.516210    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnp25\" (UniqueName: \"kubernetes.io/projected/5bceb285-968e-4147-b902-6c85c9d11fd0-kube-api-access-fnp25\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2\" (UID: \"5bceb285-968e-4147-b902-6c85c9d11fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"
Jul 31 15:36:12.516507 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.516484    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bceb285-968e-4147-b902-6c85c9d11fd0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2\" (UID: \"5bceb285-968e-4147-b902-6c85c9d11fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"
Jul 31 15:36:12.516809 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.516780    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bceb285-968e-4147-b902-6c85c9d11fd0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2\" (UID: \"5bceb285-968e-4147-b902-6c85c9d11fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"
Jul 31 15:36:12.516915 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.516786    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bceb285-968e-4147-b902-6c85c9d11fd0-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2\" (UID: \"5bceb285-968e-4147-b902-6c85c9d11fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"
Jul 31 15:36:12.518866 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.518845    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bceb285-968e-4147-b902-6c85c9d11fd0-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2\" (UID: \"5bceb285-968e-4147-b902-6c85c9d11fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"
Jul 31 15:36:12.524875 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.524845    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnp25\" (UniqueName: \"kubernetes.io/projected/5bceb285-968e-4147-b902-6c85c9d11fd0-kube-api-access-fnp25\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2\" (UID: \"5bceb285-968e-4147-b902-6c85c9d11fd0\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"
Jul 31 15:36:12.624258 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.624201    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"
Jul 31 15:36:12.641330 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.641301    2559 generic.go:356] "Generic (PLEG): container finished" podID="e44cf494-c6bd-4597-a84d-f6cb49f591e2" containerID="522cfa8f02af711319d8e938a997921676a74c1a1e41d043db0585a424a7eefa" exitCode=2
Jul 31 15:36:12.641454 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.641375    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd" event={"ID":"e44cf494-c6bd-4597-a84d-f6cb49f591e2","Type":"ContainerDied","Data":"522cfa8f02af711319d8e938a997921676a74c1a1e41d043db0585a424a7eefa"}
Jul 31 15:36:12.759673 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:12.759500    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"]
Jul 31 15:36:12.762376 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:36:12.762348    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bceb285_968e_4147_b902_6c85c9d11fd0.slice/crio-fd35e12d1a90bc4f08c9bff7e5ae056b4dca1abb155f0cdd652878d87d17c6bf WatchSource:0}: Error finding container fd35e12d1a90bc4f08c9bff7e5ae056b4dca1abb155f0cdd652878d87d17c6bf: Status 404 returned error can't find the container with id fd35e12d1a90bc4f08c9bff7e5ae056b4dca1abb155f0cdd652878d87d17c6bf
Jul 31 15:36:13.647085 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:13.647041    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2" event={"ID":"5bceb285-968e-4147-b902-6c85c9d11fd0","Type":"ContainerStarted","Data":"68246b6b41baa2773fff586fd0ce78e60f49a69c61a17ad0cb8b662b8fa748cb"}
Jul 31 15:36:13.647444 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:13.647129    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2" event={"ID":"5bceb285-968e-4147-b902-6c85c9d11fd0","Type":"ContainerStarted","Data":"fd35e12d1a90bc4f08c9bff7e5ae056b4dca1abb155f0cdd652878d87d17c6bf"}
Jul 31 15:36:14.652702 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:14.652667    2559 generic.go:356] "Generic (PLEG): container finished" podID="5bceb285-968e-4147-b902-6c85c9d11fd0" containerID="68246b6b41baa2773fff586fd0ce78e60f49a69c61a17ad0cb8b662b8fa748cb" exitCode=0
Jul 31 15:36:14.653112 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:14.652763    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2" event={"ID":"5bceb285-968e-4147-b902-6c85c9d11fd0","Type":"ContainerDied","Data":"68246b6b41baa2773fff586fd0ce78e60f49a69c61a17ad0cb8b662b8fa748cb"}
Jul 31 15:36:15.658215 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:15.658178    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2" event={"ID":"5bceb285-968e-4147-b902-6c85c9d11fd0","Type":"ContainerStarted","Data":"364c461b8fc4f56bb1dfae2d9212f29de0db8f35a7ad726b0bbf3b0d6077e064"}
Jul 31 15:36:15.658712 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:15.658235    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2" event={"ID":"5bceb285-968e-4147-b902-6c85c9d11fd0","Type":"ContainerStarted","Data":"4cb23abf59489c60a37fbedc6a5fb9ae64a48ddf8c1b0fa29fb6ae6cb42814ee"}
Jul 31 15:36:15.658712 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:15.658327    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"
Jul 31 15:36:15.680646 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:15.680586    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2" podStartSLOduration=2.62411 podStartE2EDuration="3.680569427s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.056459427s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:36:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:36:15.678599969 +0000 UTC m=+2871.990157596" watchObservedRunningTime="2026-07-31 15:36:15.680569427 +0000 UTC m=+2871.992127050"
Jul 31 15:36:16.662582 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:16.662548    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"
Jul 31 15:36:16.663696 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:16.663669    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2" podUID="5bceb285-968e-4147-b902-6c85c9d11fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 31 15:36:17.090530 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.090507    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd"
Jul 31 15:36:17.152220 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.152183    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e44cf494-c6bd-4597-a84d-f6cb49f591e2-kserve-provision-location\") pod \"e44cf494-c6bd-4597-a84d-f6cb49f591e2\" (UID: \"e44cf494-c6bd-4597-a84d-f6cb49f591e2\") "
Jul 31 15:36:17.152220 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.152224    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e44cf494-c6bd-4597-a84d-f6cb49f591e2-kube-rbac-proxy-sar-config\") pod \"e44cf494-c6bd-4597-a84d-f6cb49f591e2\" (UID: \"e44cf494-c6bd-4597-a84d-f6cb49f591e2\") "
Jul 31 15:36:17.152486 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.152257    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e44cf494-c6bd-4597-a84d-f6cb49f591e2-proxy-tls\") pod \"e44cf494-c6bd-4597-a84d-f6cb49f591e2\" (UID: \"e44cf494-c6bd-4597-a84d-f6cb49f591e2\") "
Jul 31 15:36:17.152486 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.152274    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nfp9\" (UniqueName: \"kubernetes.io/projected/e44cf494-c6bd-4597-a84d-f6cb49f591e2-kube-api-access-9nfp9\") pod \"e44cf494-c6bd-4597-a84d-f6cb49f591e2\" (UID: \"e44cf494-c6bd-4597-a84d-f6cb49f591e2\") "
Jul 31 15:36:17.152605 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.152581    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44cf494-c6bd-4597-a84d-f6cb49f591e2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e44cf494-c6bd-4597-a84d-f6cb49f591e2" (UID: "e44cf494-c6bd-4597-a84d-f6cb49f591e2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:36:17.152663 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.152608    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e44cf494-c6bd-4597-a84d-f6cb49f591e2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e44cf494-c6bd-4597-a84d-f6cb49f591e2" (UID: "e44cf494-c6bd-4597-a84d-f6cb49f591e2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:36:17.154569 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.154537    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e44cf494-c6bd-4597-a84d-f6cb49f591e2-kube-api-access-9nfp9" (OuterVolumeSpecName: "kube-api-access-9nfp9") pod "e44cf494-c6bd-4597-a84d-f6cb49f591e2" (UID: "e44cf494-c6bd-4597-a84d-f6cb49f591e2"). InnerVolumeSpecName "kube-api-access-9nfp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:36:17.154569 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.154550    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e44cf494-c6bd-4597-a84d-f6cb49f591e2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e44cf494-c6bd-4597-a84d-f6cb49f591e2" (UID: "e44cf494-c6bd-4597-a84d-f6cb49f591e2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:36:17.253272 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.253174    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e44cf494-c6bd-4597-a84d-f6cb49f591e2-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:36:17.253272 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.253215    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e44cf494-c6bd-4597-a84d-f6cb49f591e2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:36:17.253272 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.253225    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e44cf494-c6bd-4597-a84d-f6cb49f591e2-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:36:17.253272 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.253234    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nfp9\" (UniqueName: \"kubernetes.io/projected/e44cf494-c6bd-4597-a84d-f6cb49f591e2-kube-api-access-9nfp9\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:36:17.667908 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.667877    2559 generic.go:356] "Generic (PLEG): container finished" podID="e44cf494-c6bd-4597-a84d-f6cb49f591e2" containerID="97825ba273e59ec4e035d32bf499e4c1caca83c502c35d4f6de9b04eff459dd1" exitCode=0
Jul 31 15:36:17.668309 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.667973    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd"
Jul 31 15:36:17.668309 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.667976    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd" event={"ID":"e44cf494-c6bd-4597-a84d-f6cb49f591e2","Type":"ContainerDied","Data":"97825ba273e59ec4e035d32bf499e4c1caca83c502c35d4f6de9b04eff459dd1"}
Jul 31 15:36:17.668309 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.668103    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd" event={"ID":"e44cf494-c6bd-4597-a84d-f6cb49f591e2","Type":"ContainerDied","Data":"5ea29306fa5d4fc8a9ca0e1e904c87419fab75e4ad882413dc84ab5271038211"}
Jul 31 15:36:17.668309 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.668129    2559 scope.go:117] "RemoveContainer" containerID="522cfa8f02af711319d8e938a997921676a74c1a1e41d043db0585a424a7eefa"
Jul 31 15:36:17.668537 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.668514    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2" podUID="5bceb285-968e-4147-b902-6c85c9d11fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 31 15:36:17.676462 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.676445    2559 scope.go:117] "RemoveContainer" containerID="97825ba273e59ec4e035d32bf499e4c1caca83c502c35d4f6de9b04eff459dd1"
Jul 31 15:36:17.684270 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.684252    2559 scope.go:117] "RemoveContainer" containerID="72d18536008a74f6755f261ea9ae632cb5de6736080649b838fc6570719bea34"
Jul 31 15:36:17.691064 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.691043    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd"]
Jul 31 15:36:17.692005 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.691982    2559 scope.go:117] "RemoveContainer" containerID="522cfa8f02af711319d8e938a997921676a74c1a1e41d043db0585a424a7eefa"
Jul 31 15:36:17.692404 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:36:17.692283    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"522cfa8f02af711319d8e938a997921676a74c1a1e41d043db0585a424a7eefa\": container with ID starting with 522cfa8f02af711319d8e938a997921676a74c1a1e41d043db0585a424a7eefa not found: ID does not exist" containerID="522cfa8f02af711319d8e938a997921676a74c1a1e41d043db0585a424a7eefa"
Jul 31 15:36:17.692404 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.692315    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"522cfa8f02af711319d8e938a997921676a74c1a1e41d043db0585a424a7eefa"} err="failed to get container status \"522cfa8f02af711319d8e938a997921676a74c1a1e41d043db0585a424a7eefa\": rpc error: code = NotFound desc = could not find container \"522cfa8f02af711319d8e938a997921676a74c1a1e41d043db0585a424a7eefa\": container with ID starting with 522cfa8f02af711319d8e938a997921676a74c1a1e41d043db0585a424a7eefa not found: ID does not exist"
Jul 31 15:36:17.692404 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.692338    2559 scope.go:117] "RemoveContainer" containerID="97825ba273e59ec4e035d32bf499e4c1caca83c502c35d4f6de9b04eff459dd1"
Jul 31 15:36:17.692621 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:36:17.692568    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97825ba273e59ec4e035d32bf499e4c1caca83c502c35d4f6de9b04eff459dd1\": container with ID starting with 97825ba273e59ec4e035d32bf499e4c1caca83c502c35d4f6de9b04eff459dd1 not found: ID does not exist" containerID="97825ba273e59ec4e035d32bf499e4c1caca83c502c35d4f6de9b04eff459dd1"
Jul 31 15:36:17.692621 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.692586    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97825ba273e59ec4e035d32bf499e4c1caca83c502c35d4f6de9b04eff459dd1"} err="failed to get container status \"97825ba273e59ec4e035d32bf499e4c1caca83c502c35d4f6de9b04eff459dd1\": rpc error: code = NotFound desc = could not find container \"97825ba273e59ec4e035d32bf499e4c1caca83c502c35d4f6de9b04eff459dd1\": container with ID starting with 97825ba273e59ec4e035d32bf499e4c1caca83c502c35d4f6de9b04eff459dd1 not found: ID does not exist"
Jul 31 15:36:17.692621 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.692598    2559 scope.go:117] "RemoveContainer" containerID="72d18536008a74f6755f261ea9ae632cb5de6736080649b838fc6570719bea34"
Jul 31 15:36:17.692947 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:36:17.692926    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72d18536008a74f6755f261ea9ae632cb5de6736080649b838fc6570719bea34\": container with ID starting with 72d18536008a74f6755f261ea9ae632cb5de6736080649b838fc6570719bea34 not found: ID does not exist" containerID="72d18536008a74f6755f261ea9ae632cb5de6736080649b838fc6570719bea34"
Jul 31 15:36:17.693013 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.692951    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d18536008a74f6755f261ea9ae632cb5de6736080649b838fc6570719bea34"} err="failed to get container status \"72d18536008a74f6755f261ea9ae632cb5de6736080649b838fc6570719bea34\": rpc error: code = NotFound desc = could not find container \"72d18536008a74f6755f261ea9ae632cb5de6736080649b838fc6570719bea34\": container with ID starting with 72d18536008a74f6755f261ea9ae632cb5de6736080649b838fc6570719bea34 not found: ID does not exist"
Jul 31 15:36:17.693657 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:17.693636    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-567fcf85f4-snpmd"]
Jul 31 15:36:18.225361 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:18.225325    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e44cf494-c6bd-4597-a84d-f6cb49f591e2" path="/var/lib/kubelet/pods/e44cf494-c6bd-4597-a84d-f6cb49f591e2/volumes"
Jul 31 15:36:22.672670 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:22.672641    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"
Jul 31 15:36:22.673186 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:22.673162    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2" podUID="5bceb285-968e-4147-b902-6c85c9d11fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 31 15:36:32.673750 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:32.673699    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2" podUID="5bceb285-968e-4147-b902-6c85c9d11fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 31 15:36:42.673471 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:42.673419    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2" podUID="5bceb285-968e-4147-b902-6c85c9d11fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 31 15:36:52.673336 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:36:52.673294    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2" podUID="5bceb285-968e-4147-b902-6c85c9d11fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 31 15:37:02.674049 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:02.674011    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2" podUID="5bceb285-968e-4147-b902-6c85c9d11fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 31 15:37:12.673767 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:12.673704    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2" podUID="5bceb285-968e-4147-b902-6c85c9d11fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 31 15:37:22.673491 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:22.673453    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2" podUID="5bceb285-968e-4147-b902-6c85c9d11fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 31 15:37:24.218415 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:24.218361    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2" podUID="5bceb285-968e-4147-b902-6c85c9d11fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 31 15:37:34.220683 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:34.220655    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"
Jul 31 15:37:42.341297 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:42.341261    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"]
Jul 31 15:37:42.341754 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:42.341495    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2" podUID="5bceb285-968e-4147-b902-6c85c9d11fd0" containerName="kserve-container" containerID="cri-o://4cb23abf59489c60a37fbedc6a5fb9ae64a48ddf8c1b0fa29fb6ae6cb42814ee" gracePeriod=30
Jul 31 15:37:42.341754 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:42.341529    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2" podUID="5bceb285-968e-4147-b902-6c85c9d11fd0" containerName="kube-rbac-proxy" containerID="cri-o://364c461b8fc4f56bb1dfae2d9212f29de0db8f35a7ad726b0bbf3b0d6077e064" gracePeriod=30
Jul 31 15:37:42.669297 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:42.669253    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2" podUID="5bceb285-968e-4147-b902-6c85c9d11fd0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
Jul 31 15:37:42.986048 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:42.985966    2559 generic.go:356] "Generic (PLEG): container finished" podID="5bceb285-968e-4147-b902-6c85c9d11fd0" containerID="364c461b8fc4f56bb1dfae2d9212f29de0db8f35a7ad726b0bbf3b0d6077e064" exitCode=2
Jul 31 15:37:42.986048 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:42.986018    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2" event={"ID":"5bceb285-968e-4147-b902-6c85c9d11fd0","Type":"ContainerDied","Data":"364c461b8fc4f56bb1dfae2d9212f29de0db8f35a7ad726b0bbf3b0d6077e064"}
Jul 31 15:37:43.475829 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.475796    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6"]
Jul 31 15:37:43.476326 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.476102    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e44cf494-c6bd-4597-a84d-f6cb49f591e2" containerName="kube-rbac-proxy"
Jul 31 15:37:43.476326 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.476115    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e44cf494-c6bd-4597-a84d-f6cb49f591e2" containerName="kube-rbac-proxy"
Jul 31 15:37:43.476326 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.476132    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e44cf494-c6bd-4597-a84d-f6cb49f591e2" containerName="kserve-container"
Jul 31 15:37:43.476326 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.476138    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e44cf494-c6bd-4597-a84d-f6cb49f591e2" containerName="kserve-container"
Jul 31 15:37:43.476326 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.476147    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e44cf494-c6bd-4597-a84d-f6cb49f591e2" containerName="storage-initializer"
Jul 31 15:37:43.476326 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.476152    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e44cf494-c6bd-4597-a84d-f6cb49f591e2" containerName="storage-initializer"
Jul 31 15:37:43.476326 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.476214    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e44cf494-c6bd-4597-a84d-f6cb49f591e2" containerName="kserve-container"
Jul 31 15:37:43.476326 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.476223    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e44cf494-c6bd-4597-a84d-f6cb49f591e2" containerName="kube-rbac-proxy"
Jul 31 15:37:43.479178 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.479162    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6"
Jul 31 15:37:43.482251 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.482228    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Jul 31 15:37:43.482363 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.482231    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Jul 31 15:37:43.490701 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.490674    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6"]
Jul 31 15:37:43.513901 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.513871    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15fc479c-57dc-4216-a5fa-8142b8990b0a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6\" (UID: \"15fc479c-57dc-4216-a5fa-8142b8990b0a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6"
Jul 31 15:37:43.514064 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.513916    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15fc479c-57dc-4216-a5fa-8142b8990b0a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6\" (UID: \"15fc479c-57dc-4216-a5fa-8142b8990b0a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6"
Jul 31 15:37:43.514064 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.514028    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66bv2\" (UniqueName: \"kubernetes.io/projected/15fc479c-57dc-4216-a5fa-8142b8990b0a-kube-api-access-66bv2\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6\" (UID: \"15fc479c-57dc-4216-a5fa-8142b8990b0a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6"
Jul 31 15:37:43.514151 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.514073    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15fc479c-57dc-4216-a5fa-8142b8990b0a-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6\" (UID: \"15fc479c-57dc-4216-a5fa-8142b8990b0a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6"
Jul 31 15:37:43.614996 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.614954    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15fc479c-57dc-4216-a5fa-8142b8990b0a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6\" (UID: \"15fc479c-57dc-4216-a5fa-8142b8990b0a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6"
Jul 31 15:37:43.615229 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.615014    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15fc479c-57dc-4216-a5fa-8142b8990b0a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6\" (UID: \"15fc479c-57dc-4216-a5fa-8142b8990b0a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6"
Jul 31 15:37:43.615229 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.615079    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66bv2\" (UniqueName: \"kubernetes.io/projected/15fc479c-57dc-4216-a5fa-8142b8990b0a-kube-api-access-66bv2\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6\" (UID: \"15fc479c-57dc-4216-a5fa-8142b8990b0a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6"
Jul 31 15:37:43.615229 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.615119    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15fc479c-57dc-4216-a5fa-8142b8990b0a-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6\" (UID: \"15fc479c-57dc-4216-a5fa-8142b8990b0a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6"
Jul 31 15:37:43.615494 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.615465    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15fc479c-57dc-4216-a5fa-8142b8990b0a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6\" (UID: \"15fc479c-57dc-4216-a5fa-8142b8990b0a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6"
Jul 31 15:37:43.615775 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.615724    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15fc479c-57dc-4216-a5fa-8142b8990b0a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6\" (UID: \"15fc479c-57dc-4216-a5fa-8142b8990b0a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6"
Jul 31 15:37:43.617983 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.617948    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15fc479c-57dc-4216-a5fa-8142b8990b0a-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6\" (UID: \"15fc479c-57dc-4216-a5fa-8142b8990b0a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6"
Jul 31 15:37:43.628409 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.628382    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66bv2\" (UniqueName: \"kubernetes.io/projected/15fc479c-57dc-4216-a5fa-8142b8990b0a-kube-api-access-66bv2\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6\" (UID: \"15fc479c-57dc-4216-a5fa-8142b8990b0a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6"
Jul 31 15:37:43.790175 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.790086    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6"
Jul 31 15:37:43.923198 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.923163    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6"]
Jul 31 15:37:43.926292 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:37:43.926263    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15fc479c_57dc_4216_a5fa_8142b8990b0a.slice/crio-e6a942d471deabb3ecd7d0265c86f427aeedc50a9117f8b4c9b43ae4abd09512 WatchSource:0}: Error finding container e6a942d471deabb3ecd7d0265c86f427aeedc50a9117f8b4c9b43ae4abd09512: Status 404 returned error can't find the container with id e6a942d471deabb3ecd7d0265c86f427aeedc50a9117f8b4c9b43ae4abd09512
Jul 31 15:37:43.990853 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.990821    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6" event={"ID":"15fc479c-57dc-4216-a5fa-8142b8990b0a","Type":"ContainerStarted","Data":"30171c7055f9678dc49ad3d945e215dde15e8a58f344309f7cfe8d0ca2d3ce05"}
Jul 31 15:37:43.990987 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:43.990888    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6" event={"ID":"15fc479c-57dc-4216-a5fa-8142b8990b0a","Type":"ContainerStarted","Data":"e6a942d471deabb3ecd7d0265c86f427aeedc50a9117f8b4c9b43ae4abd09512"}
Jul 31 15:37:44.223122 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:44.223077    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2" podUID="5bceb285-968e-4147-b902-6c85c9d11fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 31 15:37:47.280561 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:47.280538    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"
Jul 31 15:37:47.347803 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:47.347772    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bceb285-968e-4147-b902-6c85c9d11fd0-kserve-provision-location\") pod \"5bceb285-968e-4147-b902-6c85c9d11fd0\" (UID: \"5bceb285-968e-4147-b902-6c85c9d11fd0\") "
Jul 31 15:37:47.347993 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:47.347825    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnp25\" (UniqueName: \"kubernetes.io/projected/5bceb285-968e-4147-b902-6c85c9d11fd0-kube-api-access-fnp25\") pod \"5bceb285-968e-4147-b902-6c85c9d11fd0\" (UID: \"5bceb285-968e-4147-b902-6c85c9d11fd0\") "
Jul 31 15:37:47.347993 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:47.347847    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bceb285-968e-4147-b902-6c85c9d11fd0-cabundle-cert\") pod \"5bceb285-968e-4147-b902-6c85c9d11fd0\" (UID: \"5bceb285-968e-4147-b902-6c85c9d11fd0\") "
Jul 31 15:37:47.347993 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:47.347877    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bceb285-968e-4147-b902-6c85c9d11fd0-kube-rbac-proxy-sar-config\") pod \"5bceb285-968e-4147-b902-6c85c9d11fd0\" (UID: \"5bceb285-968e-4147-b902-6c85c9d11fd0\") "
Jul 31 15:37:47.347993 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:47.347899    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bceb285-968e-4147-b902-6c85c9d11fd0-proxy-tls\") pod \"5bceb285-968e-4147-b902-6c85c9d11fd0\" (UID: \"5bceb285-968e-4147-b902-6c85c9d11fd0\") "
Jul 31 15:37:47.348223 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:47.348160    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bceb285-968e-4147-b902-6c85c9d11fd0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5bceb285-968e-4147-b902-6c85c9d11fd0" (UID: "5bceb285-968e-4147-b902-6c85c9d11fd0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:37:47.348264 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:47.348233    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bceb285-968e-4147-b902-6c85c9d11fd0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5bceb285-968e-4147-b902-6c85c9d11fd0" (UID: "5bceb285-968e-4147-b902-6c85c9d11fd0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:37:47.348303 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:47.348265    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bceb285-968e-4147-b902-6c85c9d11fd0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5bceb285-968e-4147-b902-6c85c9d11fd0" (UID: "5bceb285-968e-4147-b902-6c85c9d11fd0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:37:47.350271 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:47.350245    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bceb285-968e-4147-b902-6c85c9d11fd0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5bceb285-968e-4147-b902-6c85c9d11fd0" (UID: "5bceb285-968e-4147-b902-6c85c9d11fd0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:37:47.350345 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:47.350278    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bceb285-968e-4147-b902-6c85c9d11fd0-kube-api-access-fnp25" (OuterVolumeSpecName: "kube-api-access-fnp25") pod "5bceb285-968e-4147-b902-6c85c9d11fd0" (UID: "5bceb285-968e-4147-b902-6c85c9d11fd0"). InnerVolumeSpecName "kube-api-access-fnp25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:37:47.448373 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:47.448340    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bceb285-968e-4147-b902-6c85c9d11fd0-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:37:47.448373 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:47.448367    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnp25\" (UniqueName: \"kubernetes.io/projected/5bceb285-968e-4147-b902-6c85c9d11fd0-kube-api-access-fnp25\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:37:47.448373 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:47.448376    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5bceb285-968e-4147-b902-6c85c9d11fd0-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:37:47.448373 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:47.448385    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bceb285-968e-4147-b902-6c85c9d11fd0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:37:47.448636 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:47.448396    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bceb285-968e-4147-b902-6c85c9d11fd0-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:37:48.009438 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:48.009400    2559 generic.go:356] "Generic (PLEG): container finished" podID="5bceb285-968e-4147-b902-6c85c9d11fd0" containerID="4cb23abf59489c60a37fbedc6a5fb9ae64a48ddf8c1b0fa29fb6ae6cb42814ee" exitCode=0
Jul 31 15:37:48.009645 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:48.009469    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"
Jul 31 15:37:48.009645 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:48.009472    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2" event={"ID":"5bceb285-968e-4147-b902-6c85c9d11fd0","Type":"ContainerDied","Data":"4cb23abf59489c60a37fbedc6a5fb9ae64a48ddf8c1b0fa29fb6ae6cb42814ee"}
Jul 31 15:37:48.009645 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:48.009594    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2" event={"ID":"5bceb285-968e-4147-b902-6c85c9d11fd0","Type":"ContainerDied","Data":"fd35e12d1a90bc4f08c9bff7e5ae056b4dca1abb155f0cdd652878d87d17c6bf"}
Jul 31 15:37:48.009645 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:48.009620    2559 scope.go:117] "RemoveContainer" containerID="364c461b8fc4f56bb1dfae2d9212f29de0db8f35a7ad726b0bbf3b0d6077e064"
Jul 31 15:37:48.018358 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:48.018341    2559 scope.go:117] "RemoveContainer" containerID="4cb23abf59489c60a37fbedc6a5fb9ae64a48ddf8c1b0fa29fb6ae6cb42814ee"
Jul 31 15:37:48.025899 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:48.025865    2559 scope.go:117] "RemoveContainer" containerID="68246b6b41baa2773fff586fd0ce78e60f49a69c61a17ad0cb8b662b8fa748cb"
Jul 31 15:37:48.031962 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:48.031938    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"]
Jul 31 15:37:48.033277 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:48.033262    2559 scope.go:117] "RemoveContainer" containerID="364c461b8fc4f56bb1dfae2d9212f29de0db8f35a7ad726b0bbf3b0d6077e064"
Jul 31 15:37:48.033541 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:37:48.033522    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"364c461b8fc4f56bb1dfae2d9212f29de0db8f35a7ad726b0bbf3b0d6077e064\": container with ID starting with 364c461b8fc4f56bb1dfae2d9212f29de0db8f35a7ad726b0bbf3b0d6077e064 not found: ID does not exist" containerID="364c461b8fc4f56bb1dfae2d9212f29de0db8f35a7ad726b0bbf3b0d6077e064"
Jul 31 15:37:48.033589 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:48.033551    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"364c461b8fc4f56bb1dfae2d9212f29de0db8f35a7ad726b0bbf3b0d6077e064"} err="failed to get container status \"364c461b8fc4f56bb1dfae2d9212f29de0db8f35a7ad726b0bbf3b0d6077e064\": rpc error: code = NotFound desc = could not find container \"364c461b8fc4f56bb1dfae2d9212f29de0db8f35a7ad726b0bbf3b0d6077e064\": container with ID starting with 364c461b8fc4f56bb1dfae2d9212f29de0db8f35a7ad726b0bbf3b0d6077e064 not found: ID does not exist"
Jul 31 15:37:48.033589 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:48.033568    2559 scope.go:117] "RemoveContainer" containerID="4cb23abf59489c60a37fbedc6a5fb9ae64a48ddf8c1b0fa29fb6ae6cb42814ee"
Jul 31 15:37:48.033837 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:37:48.033819    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cb23abf59489c60a37fbedc6a5fb9ae64a48ddf8c1b0fa29fb6ae6cb42814ee\": container with ID starting with 4cb23abf59489c60a37fbedc6a5fb9ae64a48ddf8c1b0fa29fb6ae6cb42814ee not found: ID does not exist" containerID="4cb23abf59489c60a37fbedc6a5fb9ae64a48ddf8c1b0fa29fb6ae6cb42814ee"
Jul 31 15:37:48.033912 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:48.033846    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cb23abf59489c60a37fbedc6a5fb9ae64a48ddf8c1b0fa29fb6ae6cb42814ee"} err="failed to get container status \"4cb23abf59489c60a37fbedc6a5fb9ae64a48ddf8c1b0fa29fb6ae6cb42814ee\": rpc error: code = NotFound desc = could not find container \"4cb23abf59489c60a37fbedc6a5fb9ae64a48ddf8c1b0fa29fb6ae6cb42814ee\": container with ID starting with 4cb23abf59489c60a37fbedc6a5fb9ae64a48ddf8c1b0fa29fb6ae6cb42814ee not found: ID does not exist"
Jul 31 15:37:48.033912 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:48.033870    2559 scope.go:117] "RemoveContainer" containerID="68246b6b41baa2773fff586fd0ce78e60f49a69c61a17ad0cb8b662b8fa748cb"
Jul 31 15:37:48.034163 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:37:48.034144    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68246b6b41baa2773fff586fd0ce78e60f49a69c61a17ad0cb8b662b8fa748cb\": container with ID starting with 68246b6b41baa2773fff586fd0ce78e60f49a69c61a17ad0cb8b662b8fa748cb not found: ID does not exist" containerID="68246b6b41baa2773fff586fd0ce78e60f49a69c61a17ad0cb8b662b8fa748cb"
Jul 31 15:37:48.034204 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:48.034168    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68246b6b41baa2773fff586fd0ce78e60f49a69c61a17ad0cb8b662b8fa748cb"} err="failed to get container status \"68246b6b41baa2773fff586fd0ce78e60f49a69c61a17ad0cb8b662b8fa748cb\": rpc error: code = NotFound desc = could not find container \"68246b6b41baa2773fff586fd0ce78e60f49a69c61a17ad0cb8b662b8fa748cb\": container with ID starting with 68246b6b41baa2773fff586fd0ce78e60f49a69c61a17ad0cb8b662b8fa748cb not found: ID does not exist"
Jul 31 15:37:48.037776 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:48.037755    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-5d5d7cddfd-kqwz2"]
Jul 31 15:37:48.222812 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:48.222779    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bceb285-968e-4147-b902-6c85c9d11fd0" path="/var/lib/kubelet/pods/5bceb285-968e-4147-b902-6c85c9d11fd0/volumes"
Jul 31 15:37:52.031157 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:52.031127    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6_15fc479c-57dc-4216-a5fa-8142b8990b0a/storage-initializer/0.log"
Jul 31 15:37:52.031587 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:52.031170    2559 generic.go:356] "Generic (PLEG): container finished" podID="15fc479c-57dc-4216-a5fa-8142b8990b0a" containerID="30171c7055f9678dc49ad3d945e215dde15e8a58f344309f7cfe8d0ca2d3ce05" exitCode=1
Jul 31 15:37:52.031587 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:52.031246    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6" event={"ID":"15fc479c-57dc-4216-a5fa-8142b8990b0a","Type":"ContainerDied","Data":"30171c7055f9678dc49ad3d945e215dde15e8a58f344309f7cfe8d0ca2d3ce05"}
Jul 31 15:37:53.036915 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:53.036889    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6_15fc479c-57dc-4216-a5fa-8142b8990b0a/storage-initializer/0.log"
Jul 31 15:37:53.037316 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:53.036952    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6" event={"ID":"15fc479c-57dc-4216-a5fa-8142b8990b0a","Type":"ContainerStarted","Data":"1228491fec1aa40bfb391a09aa283e18133c8cdfe45b2166ebb739d5965af773"}
Jul 31 15:37:53.442713 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:53.442682    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6"]
Jul 31 15:37:54.040513 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.040478    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6" podUID="15fc479c-57dc-4216-a5fa-8142b8990b0a" containerName="storage-initializer" containerID="cri-o://1228491fec1aa40bfb391a09aa283e18133c8cdfe45b2166ebb739d5965af773" gracePeriod=30
Jul 31 15:37:54.562208 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.562170    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"]
Jul 31 15:37:54.562489 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.562477    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bceb285-968e-4147-b902-6c85c9d11fd0" containerName="kube-rbac-proxy"
Jul 31 15:37:54.562546 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.562491    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bceb285-968e-4147-b902-6c85c9d11fd0" containerName="kube-rbac-proxy"
Jul 31 15:37:54.562546 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.562500    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bceb285-968e-4147-b902-6c85c9d11fd0" containerName="storage-initializer"
Jul 31 15:37:54.562546 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.562506    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bceb285-968e-4147-b902-6c85c9d11fd0" containerName="storage-initializer"
Jul 31 15:37:54.562546 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.562519    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bceb285-968e-4147-b902-6c85c9d11fd0" containerName="kserve-container"
Jul 31 15:37:54.562546 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.562525    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bceb285-968e-4147-b902-6c85c9d11fd0" containerName="kserve-container"
Jul 31 15:37:54.562722 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.562569    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bceb285-968e-4147-b902-6c85c9d11fd0" containerName="kserve-container"
Jul 31 15:37:54.562722 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.562579    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bceb285-968e-4147-b902-6c85c9d11fd0" containerName="kube-rbac-proxy"
Jul 31 15:37:54.565499 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.565484    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"
Jul 31 15:37:54.568395 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.568370    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Jul 31 15:37:54.568522 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.568413    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Jul 31 15:37:54.568642 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.568630    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:37:54.582541 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.582515    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"]
Jul 31 15:37:54.603931 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.603898    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh\" (UID: \"7abba5d1-74d4-4b04-a2cf-48c147cb8b41\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"
Jul 31 15:37:54.604090 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.603942    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh\" (UID: \"7abba5d1-74d4-4b04-a2cf-48c147cb8b41\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"
Jul 31 15:37:54.604090 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.603960    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh\" (UID: \"7abba5d1-74d4-4b04-a2cf-48c147cb8b41\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"
Jul 31 15:37:54.604090 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.603978    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh\" (UID: \"7abba5d1-74d4-4b04-a2cf-48c147cb8b41\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"
Jul 31 15:37:54.604090 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.604061    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pc2k\" (UniqueName: \"kubernetes.io/projected/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-kube-api-access-7pc2k\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh\" (UID: \"7abba5d1-74d4-4b04-a2cf-48c147cb8b41\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"
Jul 31 15:37:54.704915 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.704879    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh\" (UID: \"7abba5d1-74d4-4b04-a2cf-48c147cb8b41\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"
Jul 31 15:37:54.705100 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.704923    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh\" (UID: \"7abba5d1-74d4-4b04-a2cf-48c147cb8b41\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"
Jul 31 15:37:54.705100 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.704941    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh\" (UID: \"7abba5d1-74d4-4b04-a2cf-48c147cb8b41\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"
Jul 31 15:37:54.705100 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.704957    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh\" (UID: \"7abba5d1-74d4-4b04-a2cf-48c147cb8b41\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"
Jul 31 15:37:54.705100 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.704981    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pc2k\" (UniqueName: \"kubernetes.io/projected/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-kube-api-access-7pc2k\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh\" (UID: \"7abba5d1-74d4-4b04-a2cf-48c147cb8b41\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"
Jul 31 15:37:54.705377 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.705352    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh\" (UID: \"7abba5d1-74d4-4b04-a2cf-48c147cb8b41\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"
Jul 31 15:37:54.705638 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.705617    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh\" (UID: \"7abba5d1-74d4-4b04-a2cf-48c147cb8b41\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"
Jul 31 15:37:54.705785 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.705676    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh\" (UID: \"7abba5d1-74d4-4b04-a2cf-48c147cb8b41\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"
Jul 31 15:37:54.707808 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.707786    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh\" (UID: \"7abba5d1-74d4-4b04-a2cf-48c147cb8b41\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"
Jul 31 15:37:54.713476 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.713454    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pc2k\" (UniqueName: \"kubernetes.io/projected/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-kube-api-access-7pc2k\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh\" (UID: \"7abba5d1-74d4-4b04-a2cf-48c147cb8b41\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"
Jul 31 15:37:54.875073 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:54.875041    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"
Jul 31 15:37:55.013584 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:55.013553    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"]
Jul 31 15:37:55.016454 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:37:55.016423    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7abba5d1_74d4_4b04_a2cf_48c147cb8b41.slice/crio-abc1d8f2139cfc108ae14b0c5b7a43d0619155ec3bf3d1dd5f3d8f2dcefe796f WatchSource:0}: Error finding container abc1d8f2139cfc108ae14b0c5b7a43d0619155ec3bf3d1dd5f3d8f2dcefe796f: Status 404 returned error can't find the container with id abc1d8f2139cfc108ae14b0c5b7a43d0619155ec3bf3d1dd5f3d8f2dcefe796f
Jul 31 15:37:55.045146 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:55.045117    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh" event={"ID":"7abba5d1-74d4-4b04-a2cf-48c147cb8b41","Type":"ContainerStarted","Data":"abc1d8f2139cfc108ae14b0c5b7a43d0619155ec3bf3d1dd5f3d8f2dcefe796f"}
Jul 31 15:37:56.050418 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:56.050379    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh" event={"ID":"7abba5d1-74d4-4b04-a2cf-48c147cb8b41","Type":"ContainerStarted","Data":"4f1567d9d6734213a5c19715f5d9eae2062f9004f5e3f45e9656ccb9319e725a"}
Jul 31 15:37:57.055477 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:57.055442    2559 generic.go:356] "Generic (PLEG): container finished" podID="7abba5d1-74d4-4b04-a2cf-48c147cb8b41" containerID="4f1567d9d6734213a5c19715f5d9eae2062f9004f5e3f45e9656ccb9319e725a" exitCode=0
Jul 31 15:37:57.055917 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:57.055525    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh" event={"ID":"7abba5d1-74d4-4b04-a2cf-48c147cb8b41","Type":"ContainerDied","Data":"4f1567d9d6734213a5c19715f5d9eae2062f9004f5e3f45e9656ccb9319e725a"}
Jul 31 15:37:58.062273 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:58.062234    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh" event={"ID":"7abba5d1-74d4-4b04-a2cf-48c147cb8b41","Type":"ContainerStarted","Data":"5b7162501f8ee295e89e4d5978fa60a5da7b2ebe049a507873470a710cb8fd2c"}
Jul 31 15:37:58.062793 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:58.062304    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh" event={"ID":"7abba5d1-74d4-4b04-a2cf-48c147cb8b41","Type":"ContainerStarted","Data":"a08fca3d23018c1f1d7db98b94eb9c4329061594a0163cd07a531baabb9770bf"}
Jul 31 15:37:58.062793 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:58.062341    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"
Jul 31 15:37:58.062793 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:58.062370    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"
Jul 31 15:37:58.063813 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:58.063788    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh" podUID="7abba5d1-74d4-4b04-a2cf-48c147cb8b41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 31 15:37:58.083380 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:58.083331    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh" podStartSLOduration=3.054228338 podStartE2EDuration="4.083314577s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.029086239s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:37:58.082084857 +0000 UTC m=+2974.393642504" watchObservedRunningTime="2026-07-31 15:37:58.083314577 +0000 UTC m=+2974.394872257"
Jul 31 15:37:58.281172 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:58.281149    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6_15fc479c-57dc-4216-a5fa-8142b8990b0a/storage-initializer/1.log"
Jul 31 15:37:58.281542 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:58.281527    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6_15fc479c-57dc-4216-a5fa-8142b8990b0a/storage-initializer/0.log"
Jul 31 15:37:58.283823 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:58.283808    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6"
Jul 31 15:37:58.333090 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:58.332996    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15fc479c-57dc-4216-a5fa-8142b8990b0a-proxy-tls\") pod \"15fc479c-57dc-4216-a5fa-8142b8990b0a\" (UID: \"15fc479c-57dc-4216-a5fa-8142b8990b0a\") "
Jul 31 15:37:58.333090 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:58.333068    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15fc479c-57dc-4216-a5fa-8142b8990b0a-kube-rbac-proxy-sar-config\") pod \"15fc479c-57dc-4216-a5fa-8142b8990b0a\" (UID: \"15fc479c-57dc-4216-a5fa-8142b8990b0a\") "
Jul 31 15:37:58.333283 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:58.333097    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15fc479c-57dc-4216-a5fa-8142b8990b0a-kserve-provision-location\") pod \"15fc479c-57dc-4216-a5fa-8142b8990b0a\" (UID: \"15fc479c-57dc-4216-a5fa-8142b8990b0a\") "
Jul 31 15:37:58.333283 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:58.333126    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66bv2\" (UniqueName: \"kubernetes.io/projected/15fc479c-57dc-4216-a5fa-8142b8990b0a-kube-api-access-66bv2\") pod \"15fc479c-57dc-4216-a5fa-8142b8990b0a\" (UID: \"15fc479c-57dc-4216-a5fa-8142b8990b0a\") "
Jul 31 15:37:58.333389 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:58.333357    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15fc479c-57dc-4216-a5fa-8142b8990b0a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "15fc479c-57dc-4216-a5fa-8142b8990b0a" (UID: "15fc479c-57dc-4216-a5fa-8142b8990b0a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:37:58.333450 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:58.333395    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15fc479c-57dc-4216-a5fa-8142b8990b0a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "15fc479c-57dc-4216-a5fa-8142b8990b0a" (UID: "15fc479c-57dc-4216-a5fa-8142b8990b0a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:37:58.333612 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:58.333592    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15fc479c-57dc-4216-a5fa-8142b8990b0a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:37:58.333612 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:58.333613    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15fc479c-57dc-4216-a5fa-8142b8990b0a-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:37:58.335335 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:58.335314    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15fc479c-57dc-4216-a5fa-8142b8990b0a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "15fc479c-57dc-4216-a5fa-8142b8990b0a" (UID: "15fc479c-57dc-4216-a5fa-8142b8990b0a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:37:58.335411 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:58.335391    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15fc479c-57dc-4216-a5fa-8142b8990b0a-kube-api-access-66bv2" (OuterVolumeSpecName: "kube-api-access-66bv2") pod "15fc479c-57dc-4216-a5fa-8142b8990b0a" (UID: "15fc479c-57dc-4216-a5fa-8142b8990b0a"). InnerVolumeSpecName "kube-api-access-66bv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:37:58.434057 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:58.434027    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15fc479c-57dc-4216-a5fa-8142b8990b0a-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:37:58.434057 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:58.434054    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66bv2\" (UniqueName: \"kubernetes.io/projected/15fc479c-57dc-4216-a5fa-8142b8990b0a-kube-api-access-66bv2\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:37:59.068064 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:59.068037    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6_15fc479c-57dc-4216-a5fa-8142b8990b0a/storage-initializer/1.log"
Jul 31 15:37:59.068524 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:59.068435    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6_15fc479c-57dc-4216-a5fa-8142b8990b0a/storage-initializer/0.log"
Jul 31 15:37:59.068524 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:59.068478    2559 generic.go:356] "Generic (PLEG): container finished" podID="15fc479c-57dc-4216-a5fa-8142b8990b0a" containerID="1228491fec1aa40bfb391a09aa283e18133c8cdfe45b2166ebb739d5965af773" exitCode=1
Jul 31 15:37:59.068524 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:59.068512    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6" event={"ID":"15fc479c-57dc-4216-a5fa-8142b8990b0a","Type":"ContainerDied","Data":"1228491fec1aa40bfb391a09aa283e18133c8cdfe45b2166ebb739d5965af773"}
Jul 31 15:37:59.068681 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:59.068557    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6"
Jul 31 15:37:59.068681 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:59.068572    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6" event={"ID":"15fc479c-57dc-4216-a5fa-8142b8990b0a","Type":"ContainerDied","Data":"e6a942d471deabb3ecd7d0265c86f427aeedc50a9117f8b4c9b43ae4abd09512"}
Jul 31 15:37:59.068681 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:59.068599    2559 scope.go:117] "RemoveContainer" containerID="1228491fec1aa40bfb391a09aa283e18133c8cdfe45b2166ebb739d5965af773"
Jul 31 15:37:59.069314 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:59.069284    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh" podUID="7abba5d1-74d4-4b04-a2cf-48c147cb8b41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 31 15:37:59.076805 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:59.076790    2559 scope.go:117] "RemoveContainer" containerID="30171c7055f9678dc49ad3d945e215dde15e8a58f344309f7cfe8d0ca2d3ce05"
Jul 31 15:37:59.084323 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:59.084309    2559 scope.go:117] "RemoveContainer" containerID="1228491fec1aa40bfb391a09aa283e18133c8cdfe45b2166ebb739d5965af773"
Jul 31 15:37:59.084567 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:37:59.084549    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1228491fec1aa40bfb391a09aa283e18133c8cdfe45b2166ebb739d5965af773\": container with ID starting with 1228491fec1aa40bfb391a09aa283e18133c8cdfe45b2166ebb739d5965af773 not found: ID does not exist" containerID="1228491fec1aa40bfb391a09aa283e18133c8cdfe45b2166ebb739d5965af773"
Jul 31 15:37:59.084641 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:59.084575    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1228491fec1aa40bfb391a09aa283e18133c8cdfe45b2166ebb739d5965af773"} err="failed to get container status \"1228491fec1aa40bfb391a09aa283e18133c8cdfe45b2166ebb739d5965af773\": rpc error: code = NotFound desc = could not find container \"1228491fec1aa40bfb391a09aa283e18133c8cdfe45b2166ebb739d5965af773\": container with ID starting with 1228491fec1aa40bfb391a09aa283e18133c8cdfe45b2166ebb739d5965af773 not found: ID does not exist"
Jul 31 15:37:59.084641 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:59.084591    2559 scope.go:117] "RemoveContainer" containerID="30171c7055f9678dc49ad3d945e215dde15e8a58f344309f7cfe8d0ca2d3ce05"
Jul 31 15:37:59.084835 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:37:59.084819    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30171c7055f9678dc49ad3d945e215dde15e8a58f344309f7cfe8d0ca2d3ce05\": container with ID starting with 30171c7055f9678dc49ad3d945e215dde15e8a58f344309f7cfe8d0ca2d3ce05 not found: ID does not exist" containerID="30171c7055f9678dc49ad3d945e215dde15e8a58f344309f7cfe8d0ca2d3ce05"
Jul 31 15:37:59.084895 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:59.084840    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30171c7055f9678dc49ad3d945e215dde15e8a58f344309f7cfe8d0ca2d3ce05"} err="failed to get container status \"30171c7055f9678dc49ad3d945e215dde15e8a58f344309f7cfe8d0ca2d3ce05\": rpc error: code = NotFound desc = could not find container \"30171c7055f9678dc49ad3d945e215dde15e8a58f344309f7cfe8d0ca2d3ce05\": container with ID starting with 30171c7055f9678dc49ad3d945e215dde15e8a58f344309f7cfe8d0ca2d3ce05 not found: ID does not exist"
Jul 31 15:37:59.106508 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:59.106485    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6"]
Jul 31 15:37:59.110226 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:37:59.110203    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-68ddb44f8-bbhg6"]
Jul 31 15:38:00.220879 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:00.220851    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15fc479c-57dc-4216-a5fa-8142b8990b0a" path="/var/lib/kubelet/pods/15fc479c-57dc-4216-a5fa-8142b8990b0a/volumes"
Jul 31 15:38:04.073601 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:04.073571    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"
Jul 31 15:38:04.074179 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:04.074152    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh" podUID="7abba5d1-74d4-4b04-a2cf-48c147cb8b41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 31 15:38:14.074600 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:14.074559    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh" podUID="7abba5d1-74d4-4b04-a2cf-48c147cb8b41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 31 15:38:24.074596 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:24.074556    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh" podUID="7abba5d1-74d4-4b04-a2cf-48c147cb8b41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 31 15:38:34.074139 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:34.074098    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh" podUID="7abba5d1-74d4-4b04-a2cf-48c147cb8b41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 31 15:38:44.074674 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:44.074634    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh" podUID="7abba5d1-74d4-4b04-a2cf-48c147cb8b41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 31 15:38:50.438562 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.438525    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"]
Jul 31 15:38:50.439032 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.438811    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15fc479c-57dc-4216-a5fa-8142b8990b0a" containerName="storage-initializer"
Jul 31 15:38:50.439032 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.438823    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="15fc479c-57dc-4216-a5fa-8142b8990b0a" containerName="storage-initializer"
Jul 31 15:38:50.439032 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.438877    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="15fc479c-57dc-4216-a5fa-8142b8990b0a" containerName="storage-initializer"
Jul 31 15:38:50.439032 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.438884    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="15fc479c-57dc-4216-a5fa-8142b8990b0a" containerName="storage-initializer"
Jul 31 15:38:50.439032 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.438934    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15fc479c-57dc-4216-a5fa-8142b8990b0a" containerName="storage-initializer"
Jul 31 15:38:50.439032 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.438941    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="15fc479c-57dc-4216-a5fa-8142b8990b0a" containerName="storage-initializer"
Jul 31 15:38:50.441872 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.441856    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"
Jul 31 15:38:50.445614 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.445592    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Jul 31 15:38:50.445780 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.445614    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 15:38:50.445780 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.445637    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Jul 31 15:38:50.455174 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.455153    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"]
Jul 31 15:38:50.536271 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.536238    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6715fd5a-c85d-41ab-8b52-a2f669616d3f-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-b65fv\" (UID: \"6715fd5a-c85d-41ab-8b52-a2f669616d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"
Jul 31 15:38:50.536444 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.536281    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6715fd5a-c85d-41ab-8b52-a2f669616d3f-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-b65fv\" (UID: \"6715fd5a-c85d-41ab-8b52-a2f669616d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"
Jul 31 15:38:50.536444 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.536309    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n246r\" (UniqueName: \"kubernetes.io/projected/6715fd5a-c85d-41ab-8b52-a2f669616d3f-kube-api-access-n246r\") pod \"isvc-pmml-predictor-8674795f59-b65fv\" (UID: \"6715fd5a-c85d-41ab-8b52-a2f669616d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"
Jul 31 15:38:50.536444 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.536326    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6715fd5a-c85d-41ab-8b52-a2f669616d3f-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-b65fv\" (UID: \"6715fd5a-c85d-41ab-8b52-a2f669616d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"
Jul 31 15:38:50.536444 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.536345    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6715fd5a-c85d-41ab-8b52-a2f669616d3f-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-b65fv\" (UID: \"6715fd5a-c85d-41ab-8b52-a2f669616d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"
Jul 31 15:38:50.636705 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.636683    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6715fd5a-c85d-41ab-8b52-a2f669616d3f-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-b65fv\" (UID: \"6715fd5a-c85d-41ab-8b52-a2f669616d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"
Jul 31 15:38:50.636818 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.636719    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6715fd5a-c85d-41ab-8b52-a2f669616d3f-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-b65fv\" (UID: \"6715fd5a-c85d-41ab-8b52-a2f669616d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"
Jul 31 15:38:50.636818 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.636756    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n246r\" (UniqueName: \"kubernetes.io/projected/6715fd5a-c85d-41ab-8b52-a2f669616d3f-kube-api-access-n246r\") pod \"isvc-pmml-predictor-8674795f59-b65fv\" (UID: \"6715fd5a-c85d-41ab-8b52-a2f669616d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"
Jul 31 15:38:50.636931 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.636873    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6715fd5a-c85d-41ab-8b52-a2f669616d3f-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-b65fv\" (UID: \"6715fd5a-c85d-41ab-8b52-a2f669616d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"
Jul 31 15:38:50.636931 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.636915    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6715fd5a-c85d-41ab-8b52-a2f669616d3f-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-b65fv\" (UID: \"6715fd5a-c85d-41ab-8b52-a2f669616d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"
Jul 31 15:38:50.637130 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.637084    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6715fd5a-c85d-41ab-8b52-a2f669616d3f-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-b65fv\" (UID: \"6715fd5a-c85d-41ab-8b52-a2f669616d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"
Jul 31 15:38:50.637326 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.637309    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6715fd5a-c85d-41ab-8b52-a2f669616d3f-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-b65fv\" (UID: \"6715fd5a-c85d-41ab-8b52-a2f669616d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"
Jul 31 15:38:50.637465 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.637447    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6715fd5a-c85d-41ab-8b52-a2f669616d3f-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-b65fv\" (UID: \"6715fd5a-c85d-41ab-8b52-a2f669616d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"
Jul 31 15:38:50.639453 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.639437    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6715fd5a-c85d-41ab-8b52-a2f669616d3f-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-b65fv\" (UID: \"6715fd5a-c85d-41ab-8b52-a2f669616d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"
Jul 31 15:38:50.646560 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.646537    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n246r\" (UniqueName: \"kubernetes.io/projected/6715fd5a-c85d-41ab-8b52-a2f669616d3f-kube-api-access-n246r\") pod \"isvc-pmml-predictor-8674795f59-b65fv\" (UID: \"6715fd5a-c85d-41ab-8b52-a2f669616d3f\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"
Jul 31 15:38:50.752259 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.752179    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"
Jul 31 15:38:50.887749 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:50.887710    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"]
Jul 31 15:38:50.890253 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:38:50.890219    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6715fd5a_c85d_41ab_8b52_a2f669616d3f.slice/crio-c68a26c3da7ca5413c9c45115e090a8399eab0219f827e090160d70d44ba0ae3 WatchSource:0}: Error finding container c68a26c3da7ca5413c9c45115e090a8399eab0219f827e090160d70d44ba0ae3: Status 404 returned error can't find the container with id c68a26c3da7ca5413c9c45115e090a8399eab0219f827e090160d70d44ba0ae3
Jul 31 15:38:51.267845 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:51.267805    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv" event={"ID":"6715fd5a-c85d-41ab-8b52-a2f669616d3f","Type":"ContainerStarted","Data":"1f739ba64fc45c289bea3888036f1bd0a0435a68ae10c86e66f708c61d1aa7ef"}
Jul 31 15:38:51.268009 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:51.267878    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv" event={"ID":"6715fd5a-c85d-41ab-8b52-a2f669616d3f","Type":"ContainerStarted","Data":"c68a26c3da7ca5413c9c45115e090a8399eab0219f827e090160d70d44ba0ae3"}
Jul 31 15:38:54.074985 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:54.074944    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh" podUID="7abba5d1-74d4-4b04-a2cf-48c147cb8b41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 31 15:38:55.285714 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:55.285632    2559 generic.go:356] "Generic (PLEG): container finished" podID="6715fd5a-c85d-41ab-8b52-a2f669616d3f" containerID="1f739ba64fc45c289bea3888036f1bd0a0435a68ae10c86e66f708c61d1aa7ef" exitCode=0
Jul 31 15:38:55.286093 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:38:55.285710    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv" event={"ID":"6715fd5a-c85d-41ab-8b52-a2f669616d3f","Type":"ContainerDied","Data":"1f739ba64fc45c289bea3888036f1bd0a0435a68ae10c86e66f708c61d1aa7ef"}
Jul 31 15:39:02.321857 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:02.321816    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv" event={"ID":"6715fd5a-c85d-41ab-8b52-a2f669616d3f","Type":"ContainerStarted","Data":"86b7c20830840ad6b97da433ea07589dc29f3249af21bfa40ab1e3241df814a1"}
Jul 31 15:39:02.322330 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:02.321889    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv" event={"ID":"6715fd5a-c85d-41ab-8b52-a2f669616d3f","Type":"ContainerStarted","Data":"b4b7f487dd028d6fad0243f77b19393bb86ba9ca1266e7125f81547b4ca74ad9"}
Jul 31 15:39:02.322330 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:02.322115    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"
Jul 31 15:39:02.352307 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:02.352251    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv" podStartSLOduration=1.615228627 podStartE2EDuration="12.35223042s" totalImagesPullingTime="6.826911017s" totalInitContainerRuntime="3.910090776s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:38:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:39:02.351836605 +0000 UTC m=+3038.663394225" watchObservedRunningTime="2026-07-31 15:39:02.35223042 +0000 UTC m=+3038.663788042"
Jul 31 15:39:03.325806 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:03.325768    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"
Jul 31 15:39:03.326965 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:03.326937    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv" podUID="6715fd5a-c85d-41ab-8b52-a2f669616d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 31 15:39:04.074680 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:04.074620    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh" podUID="7abba5d1-74d4-4b04-a2cf-48c147cb8b41" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jul 31 15:39:04.330233 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:04.330128    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv" podUID="6715fd5a-c85d-41ab-8b52-a2f669616d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 31 15:39:09.334633 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:09.334602    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"
Jul 31 15:39:09.335188 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:09.335160    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv" podUID="6715fd5a-c85d-41ab-8b52-a2f669616d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 31 15:39:14.075620 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:14.075593    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"
Jul 31 15:39:14.735325 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:14.735294    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"]
Jul 31 15:39:14.735541 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:14.735515    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh" podUID="7abba5d1-74d4-4b04-a2cf-48c147cb8b41" containerName="kserve-container" containerID="cri-o://a08fca3d23018c1f1d7db98b94eb9c4329061594a0163cd07a531baabb9770bf" gracePeriod=30
Jul 31 15:39:14.735648 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:14.735552    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh" podUID="7abba5d1-74d4-4b04-a2cf-48c147cb8b41" containerName="kube-rbac-proxy" containerID="cri-o://5b7162501f8ee295e89e4d5978fa60a5da7b2ebe049a507873470a710cb8fd2c" gracePeriod=30
Jul 31 15:39:15.374339 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:15.374306    2559 generic.go:356] "Generic (PLEG): container finished" podID="7abba5d1-74d4-4b04-a2cf-48c147cb8b41" containerID="5b7162501f8ee295e89e4d5978fa60a5da7b2ebe049a507873470a710cb8fd2c" exitCode=2
Jul 31 15:39:15.374712 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:15.374345    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh" event={"ID":"7abba5d1-74d4-4b04-a2cf-48c147cb8b41","Type":"ContainerDied","Data":"5b7162501f8ee295e89e4d5978fa60a5da7b2ebe049a507873470a710cb8fd2c"}
Jul 31 15:39:19.069912 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:19.069857    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh" podUID="7abba5d1-74d4-4b04-a2cf-48c147cb8b41" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
Jul 31 15:39:19.335591 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:19.335503    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv" podUID="6715fd5a-c85d-41ab-8b52-a2f669616d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 31 15:39:19.673327 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:19.673300    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"
Jul 31 15:39:19.756306 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:19.756267    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-proxy-tls\") pod \"7abba5d1-74d4-4b04-a2cf-48c147cb8b41\" (UID: \"7abba5d1-74d4-4b04-a2cf-48c147cb8b41\") "
Jul 31 15:39:19.756477 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:19.756315    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-kserve-provision-location\") pod \"7abba5d1-74d4-4b04-a2cf-48c147cb8b41\" (UID: \"7abba5d1-74d4-4b04-a2cf-48c147cb8b41\") "
Jul 31 15:39:19.756477 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:19.756347    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pc2k\" (UniqueName: \"kubernetes.io/projected/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-kube-api-access-7pc2k\") pod \"7abba5d1-74d4-4b04-a2cf-48c147cb8b41\" (UID: \"7abba5d1-74d4-4b04-a2cf-48c147cb8b41\") "
Jul 31 15:39:19.756477 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:19.756382    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-kube-rbac-proxy-sar-config\") pod \"7abba5d1-74d4-4b04-a2cf-48c147cb8b41\" (UID: \"7abba5d1-74d4-4b04-a2cf-48c147cb8b41\") "
Jul 31 15:39:19.756477 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:19.756426    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-cabundle-cert\") pod \"7abba5d1-74d4-4b04-a2cf-48c147cb8b41\" (UID: \"7abba5d1-74d4-4b04-a2cf-48c147cb8b41\") "
Jul 31 15:39:19.756750 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:19.756706    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7abba5d1-74d4-4b04-a2cf-48c147cb8b41" (UID: "7abba5d1-74d4-4b04-a2cf-48c147cb8b41"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:39:19.756804 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:19.756762    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7abba5d1-74d4-4b04-a2cf-48c147cb8b41" (UID: "7abba5d1-74d4-4b04-a2cf-48c147cb8b41"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:39:19.756840 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:19.756815    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7abba5d1-74d4-4b04-a2cf-48c147cb8b41" (UID: "7abba5d1-74d4-4b04-a2cf-48c147cb8b41"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:39:19.758474 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:19.758456    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7abba5d1-74d4-4b04-a2cf-48c147cb8b41" (UID: "7abba5d1-74d4-4b04-a2cf-48c147cb8b41"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:39:19.758591 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:19.758572    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-kube-api-access-7pc2k" (OuterVolumeSpecName: "kube-api-access-7pc2k") pod "7abba5d1-74d4-4b04-a2cf-48c147cb8b41" (UID: "7abba5d1-74d4-4b04-a2cf-48c147cb8b41"). InnerVolumeSpecName "kube-api-access-7pc2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:39:19.857098 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:19.857012    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:39:19.857098 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:19.857039    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:39:19.857098 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:19.857047    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:39:19.857098 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:19.857056    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pc2k\" (UniqueName: \"kubernetes.io/projected/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-kube-api-access-7pc2k\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:39:19.857098 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:19.857065    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7abba5d1-74d4-4b04-a2cf-48c147cb8b41-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:39:20.394405 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:20.394355    2559 generic.go:356] "Generic (PLEG): container finished" podID="7abba5d1-74d4-4b04-a2cf-48c147cb8b41" containerID="a08fca3d23018c1f1d7db98b94eb9c4329061594a0163cd07a531baabb9770bf" exitCode=0
Jul 31 15:39:20.394911 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:20.394443    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh" event={"ID":"7abba5d1-74d4-4b04-a2cf-48c147cb8b41","Type":"ContainerDied","Data":"a08fca3d23018c1f1d7db98b94eb9c4329061594a0163cd07a531baabb9770bf"}
Jul 31 15:39:20.394911 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:20.394459    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"
Jul 31 15:39:20.394911 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:20.394566    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh" event={"ID":"7abba5d1-74d4-4b04-a2cf-48c147cb8b41","Type":"ContainerDied","Data":"abc1d8f2139cfc108ae14b0c5b7a43d0619155ec3bf3d1dd5f3d8f2dcefe796f"}
Jul 31 15:39:20.394911 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:20.394604    2559 scope.go:117] "RemoveContainer" containerID="5b7162501f8ee295e89e4d5978fa60a5da7b2ebe049a507873470a710cb8fd2c"
Jul 31 15:39:20.402991 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:20.402972    2559 scope.go:117] "RemoveContainer" containerID="a08fca3d23018c1f1d7db98b94eb9c4329061594a0163cd07a531baabb9770bf"
Jul 31 15:39:20.410684 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:20.410667    2559 scope.go:117] "RemoveContainer" containerID="4f1567d9d6734213a5c19715f5d9eae2062f9004f5e3f45e9656ccb9319e725a"
Jul 31 15:39:20.414570 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:20.414549    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"]
Jul 31 15:39:20.418664 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:20.418643    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-544bf7954b-hvvlh"]
Jul 31 15:39:20.418923 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:20.418909    2559 scope.go:117] "RemoveContainer" containerID="5b7162501f8ee295e89e4d5978fa60a5da7b2ebe049a507873470a710cb8fd2c"
Jul 31 15:39:20.419186 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:39:20.419168    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b7162501f8ee295e89e4d5978fa60a5da7b2ebe049a507873470a710cb8fd2c\": container with ID starting with 5b7162501f8ee295e89e4d5978fa60a5da7b2ebe049a507873470a710cb8fd2c not found: ID does not exist" containerID="5b7162501f8ee295e89e4d5978fa60a5da7b2ebe049a507873470a710cb8fd2c"
Jul 31 15:39:20.419250 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:20.419193    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b7162501f8ee295e89e4d5978fa60a5da7b2ebe049a507873470a710cb8fd2c"} err="failed to get container status \"5b7162501f8ee295e89e4d5978fa60a5da7b2ebe049a507873470a710cb8fd2c\": rpc error: code = NotFound desc = could not find container \"5b7162501f8ee295e89e4d5978fa60a5da7b2ebe049a507873470a710cb8fd2c\": container with ID starting with 5b7162501f8ee295e89e4d5978fa60a5da7b2ebe049a507873470a710cb8fd2c not found: ID does not exist"
Jul 31 15:39:20.419250 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:20.419210    2559 scope.go:117] "RemoveContainer" containerID="a08fca3d23018c1f1d7db98b94eb9c4329061594a0163cd07a531baabb9770bf"
Jul 31 15:39:20.419440 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:39:20.419423    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a08fca3d23018c1f1d7db98b94eb9c4329061594a0163cd07a531baabb9770bf\": container with ID starting with a08fca3d23018c1f1d7db98b94eb9c4329061594a0163cd07a531baabb9770bf not found: ID does not exist" containerID="a08fca3d23018c1f1d7db98b94eb9c4329061594a0163cd07a531baabb9770bf"
Jul 31 15:39:20.419478 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:20.419447    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a08fca3d23018c1f1d7db98b94eb9c4329061594a0163cd07a531baabb9770bf"} err="failed to get container status \"a08fca3d23018c1f1d7db98b94eb9c4329061594a0163cd07a531baabb9770bf\": rpc error: code = NotFound desc = could not find container \"a08fca3d23018c1f1d7db98b94eb9c4329061594a0163cd07a531baabb9770bf\": container with ID starting with a08fca3d23018c1f1d7db98b94eb9c4329061594a0163cd07a531baabb9770bf not found: ID does not exist"
Jul 31 15:39:20.419478 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:20.419463    2559 scope.go:117] "RemoveContainer" containerID="4f1567d9d6734213a5c19715f5d9eae2062f9004f5e3f45e9656ccb9319e725a"
Jul 31 15:39:20.419664 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:39:20.419649    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f1567d9d6734213a5c19715f5d9eae2062f9004f5e3f45e9656ccb9319e725a\": container with ID starting with 4f1567d9d6734213a5c19715f5d9eae2062f9004f5e3f45e9656ccb9319e725a not found: ID does not exist" containerID="4f1567d9d6734213a5c19715f5d9eae2062f9004f5e3f45e9656ccb9319e725a"
Jul 31 15:39:20.419712 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:20.419668    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f1567d9d6734213a5c19715f5d9eae2062f9004f5e3f45e9656ccb9319e725a"} err="failed to get container status \"4f1567d9d6734213a5c19715f5d9eae2062f9004f5e3f45e9656ccb9319e725a\": rpc error: code = NotFound desc = could not find container \"4f1567d9d6734213a5c19715f5d9eae2062f9004f5e3f45e9656ccb9319e725a\": container with ID starting with 4f1567d9d6734213a5c19715f5d9eae2062f9004f5e3f45e9656ccb9319e725a not found: ID does not exist"
Jul 31 15:39:22.221808 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:22.221780    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7abba5d1-74d4-4b04-a2cf-48c147cb8b41" path="/var/lib/kubelet/pods/7abba5d1-74d4-4b04-a2cf-48c147cb8b41/volumes"
Jul 31 15:39:26.999276 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:26.999239    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"]
Jul 31 15:39:26.999697 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:26.999527    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7abba5d1-74d4-4b04-a2cf-48c147cb8b41" containerName="kserve-container"
Jul 31 15:39:26.999697 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:26.999539    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7abba5d1-74d4-4b04-a2cf-48c147cb8b41" containerName="kserve-container"
Jul 31 15:39:26.999697 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:26.999566    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7abba5d1-74d4-4b04-a2cf-48c147cb8b41" containerName="kube-rbac-proxy"
Jul 31 15:39:26.999697 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:26.999572    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7abba5d1-74d4-4b04-a2cf-48c147cb8b41" containerName="kube-rbac-proxy"
Jul 31 15:39:26.999697 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:26.999580    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7abba5d1-74d4-4b04-a2cf-48c147cb8b41" containerName="storage-initializer"
Jul 31 15:39:26.999697 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:26.999585    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7abba5d1-74d4-4b04-a2cf-48c147cb8b41" containerName="storage-initializer"
Jul 31 15:39:26.999697 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:26.999631    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7abba5d1-74d4-4b04-a2cf-48c147cb8b41" containerName="kube-rbac-proxy"
Jul 31 15:39:26.999697 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:26.999644    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7abba5d1-74d4-4b04-a2cf-48c147cb8b41" containerName="kserve-container"
Jul 31 15:39:27.006117 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:27.006102    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"
Jul 31 15:39:27.009292 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:27.009272    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 31 15:39:27.009696 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:27.009678    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Jul 31 15:39:27.010589 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:27.010575    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Jul 31 15:39:27.032820 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:27.032797    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"]
Jul 31 15:39:27.114807 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:27.114781    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qdfz\" (UniqueName: \"kubernetes.io/projected/6df94683-f04c-4f11-825a-12208513386d-kube-api-access-2qdfz\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq\" (UID: \"6df94683-f04c-4f11-825a-12208513386d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"
Jul 31 15:39:27.114948 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:27.114826    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6df94683-f04c-4f11-825a-12208513386d-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq\" (UID: \"6df94683-f04c-4f11-825a-12208513386d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"
Jul 31 15:39:27.114948 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:27.114852    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6df94683-f04c-4f11-825a-12208513386d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq\" (UID: \"6df94683-f04c-4f11-825a-12208513386d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"
Jul 31 15:39:27.114948 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:27.114891    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6df94683-f04c-4f11-825a-12208513386d-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq\" (UID: \"6df94683-f04c-4f11-825a-12208513386d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"
Jul 31 15:39:27.115057 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:27.114965    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6df94683-f04c-4f11-825a-12208513386d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq\" (UID: \"6df94683-f04c-4f11-825a-12208513386d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"
Jul 31 15:39:27.216068 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:27.216025    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qdfz\" (UniqueName: \"kubernetes.io/projected/6df94683-f04c-4f11-825a-12208513386d-kube-api-access-2qdfz\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq\" (UID: \"6df94683-f04c-4f11-825a-12208513386d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"
Jul 31 15:39:27.216068 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:27.216069    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6df94683-f04c-4f11-825a-12208513386d-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq\" (UID: \"6df94683-f04c-4f11-825a-12208513386d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"
Jul 31 15:39:27.216332 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:27.216085    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6df94683-f04c-4f11-825a-12208513386d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq\" (UID: \"6df94683-f04c-4f11-825a-12208513386d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"
Jul 31 15:39:27.216332 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:27.216105    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6df94683-f04c-4f11-825a-12208513386d-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq\" (UID: \"6df94683-f04c-4f11-825a-12208513386d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"
Jul 31 15:39:27.216332 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:27.216132    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6df94683-f04c-4f11-825a-12208513386d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq\" (UID: \"6df94683-f04c-4f11-825a-12208513386d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"
Jul 31 15:39:27.216578 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:27.216556    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6df94683-f04c-4f11-825a-12208513386d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq\" (UID: \"6df94683-f04c-4f11-825a-12208513386d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"
Jul 31 15:39:27.216855 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:27.216836    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6df94683-f04c-4f11-825a-12208513386d-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq\" (UID: \"6df94683-f04c-4f11-825a-12208513386d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"
Jul 31 15:39:27.216952 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:27.216893    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6df94683-f04c-4f11-825a-12208513386d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq\" (UID: \"6df94683-f04c-4f11-825a-12208513386d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"
Jul 31 15:39:27.218675 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:27.218657    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6df94683-f04c-4f11-825a-12208513386d-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq\" (UID: \"6df94683-f04c-4f11-825a-12208513386d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"
Jul 31 15:39:27.226696 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:27.226678    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qdfz\" (UniqueName: \"kubernetes.io/projected/6df94683-f04c-4f11-825a-12208513386d-kube-api-access-2qdfz\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq\" (UID: \"6df94683-f04c-4f11-825a-12208513386d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"
Jul 31 15:39:27.315429 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:27.315337    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"
Jul 31 15:39:27.451150 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:27.451076    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"]
Jul 31 15:39:27.453741 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:39:27.453695    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df94683_f04c_4f11_825a_12208513386d.slice/crio-040bb451887dc0610d437a946bd523c55e19fa721c5554013b305d3f21b5569a WatchSource:0}: Error finding container 040bb451887dc0610d437a946bd523c55e19fa721c5554013b305d3f21b5569a: Status 404 returned error can't find the container with id 040bb451887dc0610d437a946bd523c55e19fa721c5554013b305d3f21b5569a
Jul 31 15:39:28.429445 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:28.429411    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq" event={"ID":"6df94683-f04c-4f11-825a-12208513386d","Type":"ContainerStarted","Data":"f5b789ddae59647cc17dad10b4951ecc7f60a766dab50fc91074de03fca2286b"}
Jul 31 15:39:28.429870 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:28.429470    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq" event={"ID":"6df94683-f04c-4f11-825a-12208513386d","Type":"ContainerStarted","Data":"040bb451887dc0610d437a946bd523c55e19fa721c5554013b305d3f21b5569a"}
Jul 31 15:39:29.336020 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:29.335987    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv" podUID="6715fd5a-c85d-41ab-8b52-a2f669616d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 31 15:39:29.435142 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:29.435107    2559 generic.go:356] "Generic (PLEG): container finished" podID="6df94683-f04c-4f11-825a-12208513386d" containerID="f5b789ddae59647cc17dad10b4951ecc7f60a766dab50fc91074de03fca2286b" exitCode=0
Jul 31 15:39:29.435524 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:29.435188    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq" event={"ID":"6df94683-f04c-4f11-825a-12208513386d","Type":"ContainerDied","Data":"f5b789ddae59647cc17dad10b4951ecc7f60a766dab50fc91074de03fca2286b"}
Jul 31 15:39:30.440678 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:30.440648    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq" event={"ID":"6df94683-f04c-4f11-825a-12208513386d","Type":"ContainerStarted","Data":"94d75613cd400c772cdb187f857cb32ab7866830a4c654af816d482212edc7a6"}
Jul 31 15:39:30.442621 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:30.440702    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq" event={"ID":"6df94683-f04c-4f11-825a-12208513386d","Type":"ContainerStarted","Data":"ad1f5ff343b443e91737f5729ed2438969da38fa80fe165281ce6f463e311587"}
Jul 31 15:39:30.442621 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:30.440766    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"
Jul 31 15:39:30.442621 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:30.440890    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"
Jul 31 15:39:30.442621 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:30.442203    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq" podUID="6df94683-f04c-4f11-825a-12208513386d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jul 31 15:39:31.444638 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:31.444602    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq" podUID="6df94683-f04c-4f11-825a-12208513386d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jul 31 15:39:36.449757 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:36.449713    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"
Jul 31 15:39:36.450331 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:36.450308    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq" podUID="6df94683-f04c-4f11-825a-12208513386d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jul 31 15:39:36.472873 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:36.472811    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq" podStartSLOduration=9.462325021 podStartE2EDuration="10.472790837s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.010465816s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:39:30.47677187 +0000 UTC m=+3066.788329491" watchObservedRunningTime="2026-07-31 15:39:36.472790837 +0000 UTC m=+3072.784348458"
Jul 31 15:39:39.335402 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:39.335358    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv" podUID="6715fd5a-c85d-41ab-8b52-a2f669616d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 31 15:39:46.450290 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:46.450249    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq" podUID="6df94683-f04c-4f11-825a-12208513386d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jul 31 15:39:49.335985 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:49.335942    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv" podUID="6715fd5a-c85d-41ab-8b52-a2f669616d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 31 15:39:56.450890 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:56.450845    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq" podUID="6df94683-f04c-4f11-825a-12208513386d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jul 31 15:39:59.335152 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:39:59.335109    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv" podUID="6715fd5a-c85d-41ab-8b52-a2f669616d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 31 15:40:06.450292 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:06.450248    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq" podUID="6df94683-f04c-4f11-825a-12208513386d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jul 31 15:40:09.335563 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:09.335522    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv" podUID="6715fd5a-c85d-41ab-8b52-a2f669616d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 31 15:40:10.216382 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:10.216334    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv" podUID="6715fd5a-c85d-41ab-8b52-a2f669616d3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jul 31 15:40:16.450814 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:16.450771    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq" podUID="6df94683-f04c-4f11-825a-12208513386d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jul 31 15:40:20.220668 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:20.220635    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"
Jul 31 15:40:26.450571 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:26.450530    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq" podUID="6df94683-f04c-4f11-825a-12208513386d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jul 31 15:40:31.515231 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:31.515198    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"]
Jul 31 15:40:31.515672 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:31.515438    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv" podUID="6715fd5a-c85d-41ab-8b52-a2f669616d3f" containerName="kserve-container" containerID="cri-o://b4b7f487dd028d6fad0243f77b19393bb86ba9ca1266e7125f81547b4ca74ad9" gracePeriod=30
Jul 31 15:40:31.515672 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:31.515476    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv" podUID="6715fd5a-c85d-41ab-8b52-a2f669616d3f" containerName="kube-rbac-proxy" containerID="cri-o://86b7c20830840ad6b97da433ea07589dc29f3249af21bfa40ab1e3241df814a1" gracePeriod=30
Jul 31 15:40:31.666724 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:31.666688    2559 generic.go:356] "Generic (PLEG): container finished" podID="6715fd5a-c85d-41ab-8b52-a2f669616d3f" containerID="86b7c20830840ad6b97da433ea07589dc29f3249af21bfa40ab1e3241df814a1" exitCode=2
Jul 31 15:40:31.666921 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:31.666768    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv" event={"ID":"6715fd5a-c85d-41ab-8b52-a2f669616d3f","Type":"ContainerDied","Data":"86b7c20830840ad6b97da433ea07589dc29f3249af21bfa40ab1e3241df814a1"}
Jul 31 15:40:34.331188 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:34.331145    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv" podUID="6715fd5a-c85d-41ab-8b52-a2f669616d3f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.60:8643/healthz\": dial tcp 10.134.0.60:8643: connect: connection refused"
Jul 31 15:40:35.252791 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.252764    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"
Jul 31 15:40:35.333458 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.333379    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n246r\" (UniqueName: \"kubernetes.io/projected/6715fd5a-c85d-41ab-8b52-a2f669616d3f-kube-api-access-n246r\") pod \"6715fd5a-c85d-41ab-8b52-a2f669616d3f\" (UID: \"6715fd5a-c85d-41ab-8b52-a2f669616d3f\") "
Jul 31 15:40:35.333458 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.333437    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6715fd5a-c85d-41ab-8b52-a2f669616d3f-kserve-provision-location\") pod \"6715fd5a-c85d-41ab-8b52-a2f669616d3f\" (UID: \"6715fd5a-c85d-41ab-8b52-a2f669616d3f\") "
Jul 31 15:40:35.334002 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.333466    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6715fd5a-c85d-41ab-8b52-a2f669616d3f-kube-rbac-proxy-sar-config\") pod \"6715fd5a-c85d-41ab-8b52-a2f669616d3f\" (UID: \"6715fd5a-c85d-41ab-8b52-a2f669616d3f\") "
Jul 31 15:40:35.334002 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.333496    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6715fd5a-c85d-41ab-8b52-a2f669616d3f-cabundle-cert\") pod \"6715fd5a-c85d-41ab-8b52-a2f669616d3f\" (UID: \"6715fd5a-c85d-41ab-8b52-a2f669616d3f\") "
Jul 31 15:40:35.334002 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.333540    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6715fd5a-c85d-41ab-8b52-a2f669616d3f-proxy-tls\") pod \"6715fd5a-c85d-41ab-8b52-a2f669616d3f\" (UID: \"6715fd5a-c85d-41ab-8b52-a2f669616d3f\") "
Jul 31 15:40:35.334002 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.333845    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6715fd5a-c85d-41ab-8b52-a2f669616d3f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6715fd5a-c85d-41ab-8b52-a2f669616d3f" (UID: "6715fd5a-c85d-41ab-8b52-a2f669616d3f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:40:35.334002 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.333903    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6715fd5a-c85d-41ab-8b52-a2f669616d3f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6715fd5a-c85d-41ab-8b52-a2f669616d3f" (UID: "6715fd5a-c85d-41ab-8b52-a2f669616d3f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:40:35.334002 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.333911    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6715fd5a-c85d-41ab-8b52-a2f669616d3f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6715fd5a-c85d-41ab-8b52-a2f669616d3f" (UID: "6715fd5a-c85d-41ab-8b52-a2f669616d3f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:40:35.335954 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.335935    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6715fd5a-c85d-41ab-8b52-a2f669616d3f-kube-api-access-n246r" (OuterVolumeSpecName: "kube-api-access-n246r") pod "6715fd5a-c85d-41ab-8b52-a2f669616d3f" (UID: "6715fd5a-c85d-41ab-8b52-a2f669616d3f"). InnerVolumeSpecName "kube-api-access-n246r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:40:35.336039 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.336021    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6715fd5a-c85d-41ab-8b52-a2f669616d3f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6715fd5a-c85d-41ab-8b52-a2f669616d3f" (UID: "6715fd5a-c85d-41ab-8b52-a2f669616d3f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:40:35.434610 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.434560    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6715fd5a-c85d-41ab-8b52-a2f669616d3f-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:40:35.434610 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.434605    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n246r\" (UniqueName: \"kubernetes.io/projected/6715fd5a-c85d-41ab-8b52-a2f669616d3f-kube-api-access-n246r\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:40:35.434610 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.434616    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6715fd5a-c85d-41ab-8b52-a2f669616d3f-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:40:35.434610 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.434625    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6715fd5a-c85d-41ab-8b52-a2f669616d3f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:40:35.434936 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.434634    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6715fd5a-c85d-41ab-8b52-a2f669616d3f-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:40:35.683984 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.683950    2559 generic.go:356] "Generic (PLEG): container finished" podID="6715fd5a-c85d-41ab-8b52-a2f669616d3f" containerID="b4b7f487dd028d6fad0243f77b19393bb86ba9ca1266e7125f81547b4ca74ad9" exitCode=0
Jul 31 15:40:35.684180 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.683989    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv" event={"ID":"6715fd5a-c85d-41ab-8b52-a2f669616d3f","Type":"ContainerDied","Data":"b4b7f487dd028d6fad0243f77b19393bb86ba9ca1266e7125f81547b4ca74ad9"}
Jul 31 15:40:35.684180 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.684014    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"
Jul 31 15:40:35.684180 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.684049    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv" event={"ID":"6715fd5a-c85d-41ab-8b52-a2f669616d3f","Type":"ContainerDied","Data":"c68a26c3da7ca5413c9c45115e090a8399eab0219f827e090160d70d44ba0ae3"}
Jul 31 15:40:35.684180 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.684085    2559 scope.go:117] "RemoveContainer" containerID="86b7c20830840ad6b97da433ea07589dc29f3249af21bfa40ab1e3241df814a1"
Jul 31 15:40:35.692772 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.692749    2559 scope.go:117] "RemoveContainer" containerID="b4b7f487dd028d6fad0243f77b19393bb86ba9ca1266e7125f81547b4ca74ad9"
Jul 31 15:40:35.700255 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.700238    2559 scope.go:117] "RemoveContainer" containerID="1f739ba64fc45c289bea3888036f1bd0a0435a68ae10c86e66f708c61d1aa7ef"
Jul 31 15:40:35.708028 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.708010    2559 scope.go:117] "RemoveContainer" containerID="86b7c20830840ad6b97da433ea07589dc29f3249af21bfa40ab1e3241df814a1"
Jul 31 15:40:35.708283 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:40:35.708260    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86b7c20830840ad6b97da433ea07589dc29f3249af21bfa40ab1e3241df814a1\": container with ID starting with 86b7c20830840ad6b97da433ea07589dc29f3249af21bfa40ab1e3241df814a1 not found: ID does not exist" containerID="86b7c20830840ad6b97da433ea07589dc29f3249af21bfa40ab1e3241df814a1"
Jul 31 15:40:35.708361 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.708291    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86b7c20830840ad6b97da433ea07589dc29f3249af21bfa40ab1e3241df814a1"} err="failed to get container status \"86b7c20830840ad6b97da433ea07589dc29f3249af21bfa40ab1e3241df814a1\": rpc error: code = NotFound desc = could not find container \"86b7c20830840ad6b97da433ea07589dc29f3249af21bfa40ab1e3241df814a1\": container with ID starting with 86b7c20830840ad6b97da433ea07589dc29f3249af21bfa40ab1e3241df814a1 not found: ID does not exist"
Jul 31 15:40:35.708361 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.708314    2559 scope.go:117] "RemoveContainer" containerID="b4b7f487dd028d6fad0243f77b19393bb86ba9ca1266e7125f81547b4ca74ad9"
Jul 31 15:40:35.708530 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:40:35.708516    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4b7f487dd028d6fad0243f77b19393bb86ba9ca1266e7125f81547b4ca74ad9\": container with ID starting with b4b7f487dd028d6fad0243f77b19393bb86ba9ca1266e7125f81547b4ca74ad9 not found: ID does not exist" containerID="b4b7f487dd028d6fad0243f77b19393bb86ba9ca1266e7125f81547b4ca74ad9"
Jul 31 15:40:35.708576 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.708535    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4b7f487dd028d6fad0243f77b19393bb86ba9ca1266e7125f81547b4ca74ad9"} err="failed to get container status \"b4b7f487dd028d6fad0243f77b19393bb86ba9ca1266e7125f81547b4ca74ad9\": rpc error: code = NotFound desc = could not find container \"b4b7f487dd028d6fad0243f77b19393bb86ba9ca1266e7125f81547b4ca74ad9\": container with ID starting with b4b7f487dd028d6fad0243f77b19393bb86ba9ca1266e7125f81547b4ca74ad9 not found: ID does not exist"
Jul 31 15:40:35.708576 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.708548    2559 scope.go:117] "RemoveContainer" containerID="1f739ba64fc45c289bea3888036f1bd0a0435a68ae10c86e66f708c61d1aa7ef"
Jul 31 15:40:35.708724 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:40:35.708709    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f739ba64fc45c289bea3888036f1bd0a0435a68ae10c86e66f708c61d1aa7ef\": container with ID starting with 1f739ba64fc45c289bea3888036f1bd0a0435a68ae10c86e66f708c61d1aa7ef not found: ID does not exist" containerID="1f739ba64fc45c289bea3888036f1bd0a0435a68ae10c86e66f708c61d1aa7ef"
Jul 31 15:40:35.708811 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.708748    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f739ba64fc45c289bea3888036f1bd0a0435a68ae10c86e66f708c61d1aa7ef"} err="failed to get container status \"1f739ba64fc45c289bea3888036f1bd0a0435a68ae10c86e66f708c61d1aa7ef\": rpc error: code = NotFound desc = could not find container \"1f739ba64fc45c289bea3888036f1bd0a0435a68ae10c86e66f708c61d1aa7ef\": container with ID starting with 1f739ba64fc45c289bea3888036f1bd0a0435a68ae10c86e66f708c61d1aa7ef not found: ID does not exist"
Jul 31 15:40:35.711070 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.711051    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"]
Jul 31 15:40:35.714158 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:35.714136    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-b65fv"]
Jul 31 15:40:36.221431 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.221399    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6715fd5a-c85d-41ab-8b52-a2f669616d3f" path="/var/lib/kubelet/pods/6715fd5a-c85d-41ab-8b52-a2f669616d3f/volumes"
Jul 31 15:40:36.450965 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.450930    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq" podUID="6df94683-f04c-4f11-825a-12208513386d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.61:8080: connect: connection refused"
Jul 31 15:40:36.684105 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.684069    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"]
Jul 31 15:40:36.684401 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.684385    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6715fd5a-c85d-41ab-8b52-a2f669616d3f" containerName="kube-rbac-proxy"
Jul 31 15:40:36.684401 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.684403    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6715fd5a-c85d-41ab-8b52-a2f669616d3f" containerName="kube-rbac-proxy"
Jul 31 15:40:36.684514 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.684418    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6715fd5a-c85d-41ab-8b52-a2f669616d3f" containerName="kserve-container"
Jul 31 15:40:36.684514 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.684423    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6715fd5a-c85d-41ab-8b52-a2f669616d3f" containerName="kserve-container"
Jul 31 15:40:36.684514 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.684432    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6715fd5a-c85d-41ab-8b52-a2f669616d3f" containerName="storage-initializer"
Jul 31 15:40:36.684514 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.684438    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6715fd5a-c85d-41ab-8b52-a2f669616d3f" containerName="storage-initializer"
Jul 31 15:40:36.684514 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.684487    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6715fd5a-c85d-41ab-8b52-a2f669616d3f" containerName="kserve-container"
Jul 31 15:40:36.684514 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.684497    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6715fd5a-c85d-41ab-8b52-a2f669616d3f" containerName="kube-rbac-proxy"
Jul 31 15:40:36.687968 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.687935    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"
Jul 31 15:40:36.690284 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.690261    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 15:40:36.690284 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.690277    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Jul 31 15:40:36.690464 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.690261    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Jul 31 15:40:36.698579 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.698556    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"]
Jul 31 15:40:36.744995 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.744966    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-tjvlr\" (UID: \"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"
Jul 31 15:40:36.745230 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.745015    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-tjvlr\" (UID: \"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"
Jul 31 15:40:36.745230 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.745042    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq65w\" (UniqueName: \"kubernetes.io/projected/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-kube-api-access-mq65w\") pod \"isvc-pmml-runtime-predictor-885fc8777-tjvlr\" (UID: \"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"
Jul 31 15:40:36.745230 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.745137    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-tjvlr\" (UID: \"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"
Jul 31 15:40:36.745230 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.745189    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-tjvlr\" (UID: \"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"
Jul 31 15:40:36.846330 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.846295    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-tjvlr\" (UID: \"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"
Jul 31 15:40:36.846330 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.846332    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq65w\" (UniqueName: \"kubernetes.io/projected/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-kube-api-access-mq65w\") pod \"isvc-pmml-runtime-predictor-885fc8777-tjvlr\" (UID: \"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"
Jul 31 15:40:36.846597 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.846364    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-tjvlr\" (UID: \"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"
Jul 31 15:40:36.846597 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.846399    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-tjvlr\" (UID: \"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"
Jul 31 15:40:36.846597 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.846422    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-tjvlr\" (UID: \"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"
Jul 31 15:40:36.846597 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:40:36.846537    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-runtime-predictor-serving-cert: secret "isvc-pmml-runtime-predictor-serving-cert" not found
Jul 31 15:40:36.846838 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:40:36.846606    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-proxy-tls podName:eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563 nodeName:}" failed. No retries permitted until 2026-07-31 15:40:37.346585233 +0000 UTC m=+3133.658142847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-proxy-tls") pod "isvc-pmml-runtime-predictor-885fc8777-tjvlr" (UID: "eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563") : secret "isvc-pmml-runtime-predictor-serving-cert" not found
Jul 31 15:40:36.846838 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.846710    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-tjvlr\" (UID: \"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"
Jul 31 15:40:36.847070 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.847053    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-tjvlr\" (UID: \"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"
Jul 31 15:40:36.847108 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.847078    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-tjvlr\" (UID: \"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"
Jul 31 15:40:36.857702 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:36.857680    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq65w\" (UniqueName: \"kubernetes.io/projected/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-kube-api-access-mq65w\") pod \"isvc-pmml-runtime-predictor-885fc8777-tjvlr\" (UID: \"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"
Jul 31 15:40:37.350998 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:37.350957    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-tjvlr\" (UID: \"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"
Jul 31 15:40:37.353565 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:37.353540    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-tjvlr\" (UID: \"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"
Jul 31 15:40:37.598246 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:37.598200    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"
Jul 31 15:40:37.736015 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:37.735983    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"]
Jul 31 15:40:37.739126 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:40:37.739099    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef52b6b_3a2a_4cfe_9ca2_77c25ffaa563.slice/crio-d6469a054e0762b3ac17fcce599730c445201671e44e0471e7639488752d03a4 WatchSource:0}: Error finding container d6469a054e0762b3ac17fcce599730c445201671e44e0471e7639488752d03a4: Status 404 returned error can't find the container with id d6469a054e0762b3ac17fcce599730c445201671e44e0471e7639488752d03a4
Jul 31 15:40:38.699821 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:38.699782    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr" event={"ID":"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563","Type":"ContainerStarted","Data":"be06f3c7f4e4ec7eb2a01dbcf1ad16b92dafe6efdad0bf1c30d0d2e0d01037b3"}
Jul 31 15:40:38.700184 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:38.699850    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr" event={"ID":"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563","Type":"ContainerStarted","Data":"d6469a054e0762b3ac17fcce599730c445201671e44e0471e7639488752d03a4"}
Jul 31 15:40:41.711746 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:41.711703    2559 generic.go:356] "Generic (PLEG): container finished" podID="eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" containerID="be06f3c7f4e4ec7eb2a01dbcf1ad16b92dafe6efdad0bf1c30d0d2e0d01037b3" exitCode=0
Jul 31 15:40:41.712098 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:41.711765    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr" event={"ID":"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563","Type":"ContainerDied","Data":"be06f3c7f4e4ec7eb2a01dbcf1ad16b92dafe6efdad0bf1c30d0d2e0d01037b3"}
Jul 31 15:40:42.717370 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:42.717328    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr" event={"ID":"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563","Type":"ContainerStarted","Data":"bdcf499f9b2c4e15cad97869a9302eaf4256db396573d48546a330f780281fdd"}
Jul 31 15:40:42.717802 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:42.717393    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr" event={"ID":"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563","Type":"ContainerStarted","Data":"2a15d2578a6649616012cf5dc522acd7f84e1c7f28940f76e75d1510b1e89414"}
Jul 31 15:40:42.717802 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:42.717668    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"
Jul 31 15:40:42.717890 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:42.717849    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"
Jul 31 15:40:42.719022 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:42.718998    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr" podUID="eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jul 31 15:40:42.739058 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:42.739005    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr" podStartSLOduration=2.840402711 podStartE2EDuration="6.738987954s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.898585243s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:40:42.737258151 +0000 UTC m=+3139.048815778" watchObservedRunningTime="2026-07-31 15:40:42.738987954 +0000 UTC m=+3139.050545574"
Jul 31 15:40:43.721489 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:43.721440    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr" podUID="eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jul 31 15:40:46.450951 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:46.450868    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"
Jul 31 15:40:46.890000 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:46.889962    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"]
Jul 31 15:40:46.890332 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:46.890306    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq" podUID="6df94683-f04c-4f11-825a-12208513386d" containerName="kserve-container" containerID="cri-o://ad1f5ff343b443e91737f5729ed2438969da38fa80fe165281ce6f463e311587" gracePeriod=30
Jul 31 15:40:46.890437 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:46.890310    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq" podUID="6df94683-f04c-4f11-825a-12208513386d" containerName="kube-rbac-proxy" containerID="cri-o://94d75613cd400c772cdb187f857cb32ab7866830a4c654af816d482212edc7a6" gracePeriod=30
Jul 31 15:40:47.737321 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:47.737288    2559 generic.go:356] "Generic (PLEG): container finished" podID="6df94683-f04c-4f11-825a-12208513386d" containerID="94d75613cd400c772cdb187f857cb32ab7866830a4c654af816d482212edc7a6" exitCode=2
Jul 31 15:40:47.737700 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:47.737337    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq" event={"ID":"6df94683-f04c-4f11-825a-12208513386d","Type":"ContainerDied","Data":"94d75613cd400c772cdb187f857cb32ab7866830a4c654af816d482212edc7a6"}
Jul 31 15:40:48.725851 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:48.725821    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"
Jul 31 15:40:48.726395 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:48.726366    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr" podUID="eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jul 31 15:40:51.445812 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:51.445763    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq" podUID="6df94683-f04c-4f11-825a-12208513386d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.61:8643/healthz\": dial tcp 10.134.0.61:8643: connect: connection refused"
Jul 31 15:40:51.755292 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:51.755261    2559 generic.go:356] "Generic (PLEG): container finished" podID="6df94683-f04c-4f11-825a-12208513386d" containerID="ad1f5ff343b443e91737f5729ed2438969da38fa80fe165281ce6f463e311587" exitCode=0
Jul 31 15:40:51.755460 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:51.755319    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq" event={"ID":"6df94683-f04c-4f11-825a-12208513386d","Type":"ContainerDied","Data":"ad1f5ff343b443e91737f5729ed2438969da38fa80fe165281ce6f463e311587"}
Jul 31 15:40:51.836978 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:51.836951    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"
Jul 31 15:40:51.969022 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:51.968910    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6df94683-f04c-4f11-825a-12208513386d-proxy-tls\") pod \"6df94683-f04c-4f11-825a-12208513386d\" (UID: \"6df94683-f04c-4f11-825a-12208513386d\") "
Jul 31 15:40:51.969022 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:51.968974    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6df94683-f04c-4f11-825a-12208513386d-cabundle-cert\") pod \"6df94683-f04c-4f11-825a-12208513386d\" (UID: \"6df94683-f04c-4f11-825a-12208513386d\") "
Jul 31 15:40:51.969022 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:51.969004    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6df94683-f04c-4f11-825a-12208513386d-kserve-provision-location\") pod \"6df94683-f04c-4f11-825a-12208513386d\" (UID: \"6df94683-f04c-4f11-825a-12208513386d\") "
Jul 31 15:40:51.969022 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:51.969024    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qdfz\" (UniqueName: \"kubernetes.io/projected/6df94683-f04c-4f11-825a-12208513386d-kube-api-access-2qdfz\") pod \"6df94683-f04c-4f11-825a-12208513386d\" (UID: \"6df94683-f04c-4f11-825a-12208513386d\") "
Jul 31 15:40:51.969388 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:51.969088    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6df94683-f04c-4f11-825a-12208513386d-kube-rbac-proxy-sar-config\") pod \"6df94683-f04c-4f11-825a-12208513386d\" (UID: \"6df94683-f04c-4f11-825a-12208513386d\") "
Jul 31 15:40:51.969446 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:51.969398    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df94683-f04c-4f11-825a-12208513386d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6df94683-f04c-4f11-825a-12208513386d" (UID: "6df94683-f04c-4f11-825a-12208513386d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:40:51.969446 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:51.969423    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df94683-f04c-4f11-825a-12208513386d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6df94683-f04c-4f11-825a-12208513386d" (UID: "6df94683-f04c-4f11-825a-12208513386d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:40:51.969548 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:51.969480    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df94683-f04c-4f11-825a-12208513386d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6df94683-f04c-4f11-825a-12208513386d" (UID: "6df94683-f04c-4f11-825a-12208513386d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:40:51.971301 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:51.971275    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df94683-f04c-4f11-825a-12208513386d-kube-api-access-2qdfz" (OuterVolumeSpecName: "kube-api-access-2qdfz") pod "6df94683-f04c-4f11-825a-12208513386d" (UID: "6df94683-f04c-4f11-825a-12208513386d"). InnerVolumeSpecName "kube-api-access-2qdfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:40:51.971427 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:51.971312    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6df94683-f04c-4f11-825a-12208513386d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6df94683-f04c-4f11-825a-12208513386d" (UID: "6df94683-f04c-4f11-825a-12208513386d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:40:52.069960 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:52.069909    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6df94683-f04c-4f11-825a-12208513386d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:40:52.069960 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:52.069949    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6df94683-f04c-4f11-825a-12208513386d-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:40:52.069960 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:52.069960    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6df94683-f04c-4f11-825a-12208513386d-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:40:52.069960 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:52.069969    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6df94683-f04c-4f11-825a-12208513386d-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:40:52.069960 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:52.069978    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qdfz\" (UniqueName: \"kubernetes.io/projected/6df94683-f04c-4f11-825a-12208513386d-kube-api-access-2qdfz\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:40:52.761409 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:52.761363    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq" event={"ID":"6df94683-f04c-4f11-825a-12208513386d","Type":"ContainerDied","Data":"040bb451887dc0610d437a946bd523c55e19fa721c5554013b305d3f21b5569a"}
Jul 31 15:40:52.761847 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:52.761419    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"
Jul 31 15:40:52.761847 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:52.761489    2559 scope.go:117] "RemoveContainer" containerID="94d75613cd400c772cdb187f857cb32ab7866830a4c654af816d482212edc7a6"
Jul 31 15:40:52.771108 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:52.771085    2559 scope.go:117] "RemoveContainer" containerID="ad1f5ff343b443e91737f5729ed2438969da38fa80fe165281ce6f463e311587"
Jul 31 15:40:52.779451 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:52.779421    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"]
Jul 31 15:40:52.779885 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:52.779869    2559 scope.go:117] "RemoveContainer" containerID="f5b789ddae59647cc17dad10b4951ecc7f60a766dab50fc91074de03fca2286b"
Jul 31 15:40:52.783018 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:52.782997    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9c5489d65-9qvjq"]
Jul 31 15:40:54.220977 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:54.220936    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6df94683-f04c-4f11-825a-12208513386d" path="/var/lib/kubelet/pods/6df94683-f04c-4f11-825a-12208513386d/volumes"
Jul 31 15:40:58.726786 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:40:58.726753    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr" podUID="eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jul 31 15:41:08.726814 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:41:08.726772    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr" podUID="eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jul 31 15:41:18.726641 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:41:18.726595    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr" podUID="eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jul 31 15:41:28.726921 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:41:28.726875    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr" podUID="eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jul 31 15:41:38.726267 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:41:38.726212    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr" podUID="eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jul 31 15:41:48.726886 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:41:48.726847    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr" podUID="eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jul 31 15:41:58.727343 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:41:58.727303    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"
Jul 31 15:42:07.922886 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:07.922853    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"]
Jul 31 15:42:07.923367 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:07.923122    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr" podUID="eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" containerName="kube-rbac-proxy" containerID="cri-o://bdcf499f9b2c4e15cad97869a9302eaf4256db396573d48546a330f780281fdd" gracePeriod=30
Jul 31 15:42:07.923367 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:07.923121    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr" podUID="eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" containerName="kserve-container" containerID="cri-o://2a15d2578a6649616012cf5dc522acd7f84e1c7f28940f76e75d1510b1e89414" gracePeriod=30
Jul 31 15:42:08.721746 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:08.721697    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr" podUID="eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.62:8643/healthz\": dial tcp 10.134.0.62:8643: connect: connection refused"
Jul 31 15:42:08.727150 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:08.727087    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr" podUID="eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jul 31 15:42:09.042463 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:09.042384    2559 generic.go:356] "Generic (PLEG): container finished" podID="eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" containerID="bdcf499f9b2c4e15cad97869a9302eaf4256db396573d48546a330f780281fdd" exitCode=2
Jul 31 15:42:09.042463 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:09.042419    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr" event={"ID":"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563","Type":"ContainerDied","Data":"bdcf499f9b2c4e15cad97869a9302eaf4256db396573d48546a330f780281fdd"}
Jul 31 15:42:11.668300 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:11.668272    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"
Jul 31 15:42:11.755105 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:11.755008    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-cabundle-cert\") pod \"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563\" (UID: \"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563\") "
Jul 31 15:42:11.755105 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:11.755046    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-kube-rbac-proxy-sar-config\") pod \"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563\" (UID: \"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563\") "
Jul 31 15:42:11.755105 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:11.755072    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-kserve-provision-location\") pod \"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563\" (UID: \"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563\") "
Jul 31 15:42:11.755468 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:11.755124    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mq65w\" (UniqueName: \"kubernetes.io/projected/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-kube-api-access-mq65w\") pod \"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563\" (UID: \"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563\") "
Jul 31 15:42:11.755468 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:11.755189    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-proxy-tls\") pod \"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563\" (UID: \"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563\") "
Jul 31 15:42:11.755468 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:11.755432    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" (UID: "eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:42:11.755468 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:11.755448    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" (UID: "eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:42:11.755614 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:11.755474    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" (UID: "eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:42:11.757408 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:11.757378    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-kube-api-access-mq65w" (OuterVolumeSpecName: "kube-api-access-mq65w") pod "eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" (UID: "eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563"). InnerVolumeSpecName "kube-api-access-mq65w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:42:11.757408 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:11.757390    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" (UID: "eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:42:11.856496 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:11.856446    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-proxy-tls\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:42:11.856496 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:11.856492    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-cabundle-cert\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:42:11.856496 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:11.856503    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:42:11.856496 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:11.856512    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-kserve-provision-location\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:42:11.856826 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:11.856521    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq65w\" (UniqueName: \"kubernetes.io/projected/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563-kube-api-access-mq65w\") on node \"ip-10-0-142-219.ec2.internal\" DevicePath \"\""
Jul 31 15:42:12.055797 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:12.055696    2559 generic.go:356] "Generic (PLEG): container finished" podID="eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" containerID="2a15d2578a6649616012cf5dc522acd7f84e1c7f28940f76e75d1510b1e89414" exitCode=0
Jul 31 15:42:12.055797 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:12.055789    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"
Jul 31 15:42:12.055989 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:12.055789    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr" event={"ID":"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563","Type":"ContainerDied","Data":"2a15d2578a6649616012cf5dc522acd7f84e1c7f28940f76e75d1510b1e89414"}
Jul 31 15:42:12.055989 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:12.055863    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr" event={"ID":"eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563","Type":"ContainerDied","Data":"d6469a054e0762b3ac17fcce599730c445201671e44e0471e7639488752d03a4"}
Jul 31 15:42:12.055989 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:12.055903    2559 scope.go:117] "RemoveContainer" containerID="bdcf499f9b2c4e15cad97869a9302eaf4256db396573d48546a330f780281fdd"
Jul 31 15:42:12.064467 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:12.064447    2559 scope.go:117] "RemoveContainer" containerID="2a15d2578a6649616012cf5dc522acd7f84e1c7f28940f76e75d1510b1e89414"
Jul 31 15:42:12.072152 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:12.072133    2559 scope.go:117] "RemoveContainer" containerID="be06f3c7f4e4ec7eb2a01dbcf1ad16b92dafe6efdad0bf1c30d0d2e0d01037b3"
Jul 31 15:42:12.077082 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:12.077054    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"]
Jul 31 15:42:12.080450 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:12.080426    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-tjvlr"]
Jul 31 15:42:12.080915 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:12.080898    2559 scope.go:117] "RemoveContainer" containerID="bdcf499f9b2c4e15cad97869a9302eaf4256db396573d48546a330f780281fdd"
Jul 31 15:42:12.081194 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:42:12.081177    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdcf499f9b2c4e15cad97869a9302eaf4256db396573d48546a330f780281fdd\": container with ID starting with bdcf499f9b2c4e15cad97869a9302eaf4256db396573d48546a330f780281fdd not found: ID does not exist" containerID="bdcf499f9b2c4e15cad97869a9302eaf4256db396573d48546a330f780281fdd"
Jul 31 15:42:12.081243 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:12.081202    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdcf499f9b2c4e15cad97869a9302eaf4256db396573d48546a330f780281fdd"} err="failed to get container status \"bdcf499f9b2c4e15cad97869a9302eaf4256db396573d48546a330f780281fdd\": rpc error: code = NotFound desc = could not find container \"bdcf499f9b2c4e15cad97869a9302eaf4256db396573d48546a330f780281fdd\": container with ID starting with bdcf499f9b2c4e15cad97869a9302eaf4256db396573d48546a330f780281fdd not found: ID does not exist"
Jul 31 15:42:12.081243 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:12.081220    2559 scope.go:117] "RemoveContainer" containerID="2a15d2578a6649616012cf5dc522acd7f84e1c7f28940f76e75d1510b1e89414"
Jul 31 15:42:12.081468 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:42:12.081452    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a15d2578a6649616012cf5dc522acd7f84e1c7f28940f76e75d1510b1e89414\": container with ID starting with 2a15d2578a6649616012cf5dc522acd7f84e1c7f28940f76e75d1510b1e89414 not found: ID does not exist" containerID="2a15d2578a6649616012cf5dc522acd7f84e1c7f28940f76e75d1510b1e89414"
Jul 31 15:42:12.081509 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:12.081478    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a15d2578a6649616012cf5dc522acd7f84e1c7f28940f76e75d1510b1e89414"} err="failed to get container status \"2a15d2578a6649616012cf5dc522acd7f84e1c7f28940f76e75d1510b1e89414\": rpc error: code = NotFound desc = could not find container \"2a15d2578a6649616012cf5dc522acd7f84e1c7f28940f76e75d1510b1e89414\": container with ID starting with 2a15d2578a6649616012cf5dc522acd7f84e1c7f28940f76e75d1510b1e89414 not found: ID does not exist"
Jul 31 15:42:12.081509 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:12.081495    2559 scope.go:117] "RemoveContainer" containerID="be06f3c7f4e4ec7eb2a01dbcf1ad16b92dafe6efdad0bf1c30d0d2e0d01037b3"
Jul 31 15:42:12.081793 ip-10-0-142-219 kubenswrapper[2559]: E0731 15:42:12.081768    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be06f3c7f4e4ec7eb2a01dbcf1ad16b92dafe6efdad0bf1c30d0d2e0d01037b3\": container with ID starting with be06f3c7f4e4ec7eb2a01dbcf1ad16b92dafe6efdad0bf1c30d0d2e0d01037b3 not found: ID does not exist" containerID="be06f3c7f4e4ec7eb2a01dbcf1ad16b92dafe6efdad0bf1c30d0d2e0d01037b3"
Jul 31 15:42:12.081857 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:12.081794    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be06f3c7f4e4ec7eb2a01dbcf1ad16b92dafe6efdad0bf1c30d0d2e0d01037b3"} err="failed to get container status \"be06f3c7f4e4ec7eb2a01dbcf1ad16b92dafe6efdad0bf1c30d0d2e0d01037b3\": rpc error: code = NotFound desc = could not find container \"be06f3c7f4e4ec7eb2a01dbcf1ad16b92dafe6efdad0bf1c30d0d2e0d01037b3\": container with ID starting with be06f3c7f4e4ec7eb2a01dbcf1ad16b92dafe6efdad0bf1c30d0d2e0d01037b3 not found: ID does not exist"
Jul 31 15:42:12.221301 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:12.221270    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" path="/var/lib/kubelet/pods/eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563/volumes"
Jul 31 15:42:50.035460 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.035415    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cxx9f/must-gather-vx22v"]
Jul 31 15:42:50.036115 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.035796    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" containerName="storage-initializer"
Jul 31 15:42:50.036115 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.035814    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" containerName="storage-initializer"
Jul 31 15:42:50.036115 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.035829    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" containerName="kube-rbac-proxy"
Jul 31 15:42:50.036115 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.035838    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" containerName="kube-rbac-proxy"
Jul 31 15:42:50.036115 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.035852    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6df94683-f04c-4f11-825a-12208513386d" containerName="storage-initializer"
Jul 31 15:42:50.036115 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.035860    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6df94683-f04c-4f11-825a-12208513386d" containerName="storage-initializer"
Jul 31 15:42:50.036115 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.035875    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" containerName="kserve-container"
Jul 31 15:42:50.036115 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.035882    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" containerName="kserve-container"
Jul 31 15:42:50.036115 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.035903    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6df94683-f04c-4f11-825a-12208513386d" containerName="kserve-container"
Jul 31 15:42:50.036115 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.035910    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6df94683-f04c-4f11-825a-12208513386d" containerName="kserve-container"
Jul 31 15:42:50.036115 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.035923    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6df94683-f04c-4f11-825a-12208513386d" containerName="kube-rbac-proxy"
Jul 31 15:42:50.036115 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.035931    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6df94683-f04c-4f11-825a-12208513386d" containerName="kube-rbac-proxy"
Jul 31 15:42:50.036115 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.036003    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6df94683-f04c-4f11-825a-12208513386d" containerName="kube-rbac-proxy"
Jul 31 15:42:50.036115 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.036016    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" containerName="kube-rbac-proxy"
Jul 31 15:42:50.036115 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.036025    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6df94683-f04c-4f11-825a-12208513386d" containerName="kserve-container"
Jul 31 15:42:50.036115 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.036033    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="eef52b6b-3a2a-4cfe-9ca2-77c25ffaa563" containerName="kserve-container"
Jul 31 15:42:50.039225 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.039204    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cxx9f/must-gather-vx22v"
Jul 31 15:42:50.041582 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.041552    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cxx9f\"/\"default-dockercfg-rzbd9\""
Jul 31 15:42:50.041582 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.041573    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cxx9f\"/\"kube-root-ca.crt\""
Jul 31 15:42:50.041781 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.041625    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cxx9f\"/\"openshift-service-ca.crt\""
Jul 31 15:42:50.047552 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.047528    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cxx9f/must-gather-vx22v"]
Jul 31 15:42:50.135882 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.135844    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbhpp\" (UniqueName: \"kubernetes.io/projected/281901f0-c71f-46a4-8fb3-c3a0ef3f6b33-kube-api-access-bbhpp\") pod \"must-gather-vx22v\" (UID: \"281901f0-c71f-46a4-8fb3-c3a0ef3f6b33\") " pod="openshift-must-gather-cxx9f/must-gather-vx22v"
Jul 31 15:42:50.136058 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.135893    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/281901f0-c71f-46a4-8fb3-c3a0ef3f6b33-must-gather-output\") pod \"must-gather-vx22v\" (UID: \"281901f0-c71f-46a4-8fb3-c3a0ef3f6b33\") " pod="openshift-must-gather-cxx9f/must-gather-vx22v"
Jul 31 15:42:50.236410 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.236369    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/281901f0-c71f-46a4-8fb3-c3a0ef3f6b33-must-gather-output\") pod \"must-gather-vx22v\" (UID: \"281901f0-c71f-46a4-8fb3-c3a0ef3f6b33\") " pod="openshift-must-gather-cxx9f/must-gather-vx22v"
Jul 31 15:42:50.236589 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.236469    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbhpp\" (UniqueName: \"kubernetes.io/projected/281901f0-c71f-46a4-8fb3-c3a0ef3f6b33-kube-api-access-bbhpp\") pod \"must-gather-vx22v\" (UID: \"281901f0-c71f-46a4-8fb3-c3a0ef3f6b33\") " pod="openshift-must-gather-cxx9f/must-gather-vx22v"
Jul 31 15:42:50.236816 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.236794    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/281901f0-c71f-46a4-8fb3-c3a0ef3f6b33-must-gather-output\") pod \"must-gather-vx22v\" (UID: \"281901f0-c71f-46a4-8fb3-c3a0ef3f6b33\") " pod="openshift-must-gather-cxx9f/must-gather-vx22v"
Jul 31 15:42:50.245283 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.245258    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbhpp\" (UniqueName: \"kubernetes.io/projected/281901f0-c71f-46a4-8fb3-c3a0ef3f6b33-kube-api-access-bbhpp\") pod \"must-gather-vx22v\" (UID: \"281901f0-c71f-46a4-8fb3-c3a0ef3f6b33\") " pod="openshift-must-gather-cxx9f/must-gather-vx22v"
Jul 31 15:42:50.348925 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.348899    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cxx9f/must-gather-vx22v"
Jul 31 15:42:50.480021 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.479849    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cxx9f/must-gather-vx22v"]
Jul 31 15:42:50.482654 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:42:50.482626    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod281901f0_c71f_46a4_8fb3_c3a0ef3f6b33.slice/crio-ed97802c653bb017176c6a1a76dbdda9b455aefce81fc7e4d686fddecf130765 WatchSource:0}: Error finding container ed97802c653bb017176c6a1a76dbdda9b455aefce81fc7e4d686fddecf130765: Status 404 returned error can't find the container with id ed97802c653bb017176c6a1a76dbdda9b455aefce81fc7e4d686fddecf130765
Jul 31 15:42:50.484498 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:50.484477    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 15:42:51.204555 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:51.204513    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cxx9f/must-gather-vx22v" event={"ID":"281901f0-c71f-46a4-8fb3-c3a0ef3f6b33","Type":"ContainerStarted","Data":"ed97802c653bb017176c6a1a76dbdda9b455aefce81fc7e4d686fddecf130765"}
Jul 31 15:42:52.211900 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:52.211865    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cxx9f/must-gather-vx22v" event={"ID":"281901f0-c71f-46a4-8fb3-c3a0ef3f6b33","Type":"ContainerStarted","Data":"fa08e0ebe45ef3200fd058dc8425abfcd53d1ca76c851493c988ada63737e77e"}
Jul 31 15:42:52.212320 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:52.211911    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cxx9f/must-gather-vx22v" event={"ID":"281901f0-c71f-46a4-8fb3-c3a0ef3f6b33","Type":"ContainerStarted","Data":"6e8a5e01dd7933b2a3c4926f213a59c9f3c0f519aba0f3249aeaa80d477a9e49"}
Jul 31 15:42:52.228764 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:52.228681    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cxx9f/must-gather-vx22v" podStartSLOduration=1.551074624 podStartE2EDuration="2.228659672s" totalImagesPullingTime="677.585048ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:42:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:42:52.228402337 +0000 UTC m=+3268.539959960" watchObservedRunningTime="2026-07-31 15:42:52.228659672 +0000 UTC m=+3268.540217295"
Jul 31 15:42:52.797466 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:52.797427    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nk8pw_d947335a-f9d7-4fcd-b9e1-457477ab7f72/global-pull-secret-syncer/0.log"
Jul 31 15:42:52.924464 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:52.924432    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xg6w7_988269ba-5182-484e-ad71-c72e50c2fe3b/konnectivity-agent/0.log"
Jul 31 15:42:52.997179 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:52.997136    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-219.ec2.internal_13814aefbbec7fd254d31d3cf317ab12/haproxy/0.log"
Jul 31 15:42:57.590294 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:57.590259    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-m4hkx_e1cfecbe-4155-4b24-aad5-2277fd0c350e/cluster-monitoring-operator/0.log"
Jul 31 15:42:57.955843 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:57.955812    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qss7j_59378969-47f3-45bd-b6f5-0407ae63516c/node-exporter/0.log"
Jul 31 15:42:57.980422 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:57.980391    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qss7j_59378969-47f3-45bd-b6f5-0407ae63516c/kube-rbac-proxy/0.log"
Jul 31 15:42:58.004255 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:42:58.004230    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qss7j_59378969-47f3-45bd-b6f5-0407ae63516c/init-textfile/0.log"
Jul 31 15:43:01.052839 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:01.052806    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cxx9f/perf-node-gather-daemonset-pm2kb"]
Jul 31 15:43:01.057978 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:01.057949    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-pm2kb"
Jul 31 15:43:01.066500 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:01.066469    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cxx9f/perf-node-gather-daemonset-pm2kb"]
Jul 31 15:43:01.128465 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:01.128422    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/767a2f9e-0f22-407f-8c1e-4f12f2242f9e-lib-modules\") pod \"perf-node-gather-daemonset-pm2kb\" (UID: \"767a2f9e-0f22-407f-8c1e-4f12f2242f9e\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-pm2kb"
Jul 31 15:43:01.128631 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:01.128481    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/767a2f9e-0f22-407f-8c1e-4f12f2242f9e-proc\") pod \"perf-node-gather-daemonset-pm2kb\" (UID: \"767a2f9e-0f22-407f-8c1e-4f12f2242f9e\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-pm2kb"
Jul 31 15:43:01.128631 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:01.128541    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/767a2f9e-0f22-407f-8c1e-4f12f2242f9e-podres\") pod \"perf-node-gather-daemonset-pm2kb\" (UID: \"767a2f9e-0f22-407f-8c1e-4f12f2242f9e\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-pm2kb"
Jul 31 15:43:01.128631 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:01.128591    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/767a2f9e-0f22-407f-8c1e-4f12f2242f9e-sys\") pod \"perf-node-gather-daemonset-pm2kb\" (UID: \"767a2f9e-0f22-407f-8c1e-4f12f2242f9e\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-pm2kb"
Jul 31 15:43:01.128771 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:01.128637    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r79jz\" (UniqueName: \"kubernetes.io/projected/767a2f9e-0f22-407f-8c1e-4f12f2242f9e-kube-api-access-r79jz\") pod \"perf-node-gather-daemonset-pm2kb\" (UID: \"767a2f9e-0f22-407f-8c1e-4f12f2242f9e\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-pm2kb"
Jul 31 15:43:01.229149 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:01.229094    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/767a2f9e-0f22-407f-8c1e-4f12f2242f9e-podres\") pod \"perf-node-gather-daemonset-pm2kb\" (UID: \"767a2f9e-0f22-407f-8c1e-4f12f2242f9e\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-pm2kb"
Jul 31 15:43:01.229354 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:01.229162    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/767a2f9e-0f22-407f-8c1e-4f12f2242f9e-sys\") pod \"perf-node-gather-daemonset-pm2kb\" (UID: \"767a2f9e-0f22-407f-8c1e-4f12f2242f9e\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-pm2kb"
Jul 31 15:43:01.229354 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:01.229189    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r79jz\" (UniqueName: \"kubernetes.io/projected/767a2f9e-0f22-407f-8c1e-4f12f2242f9e-kube-api-access-r79jz\") pod \"perf-node-gather-daemonset-pm2kb\" (UID: \"767a2f9e-0f22-407f-8c1e-4f12f2242f9e\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-pm2kb"
Jul 31 15:43:01.229354 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:01.229250    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/767a2f9e-0f22-407f-8c1e-4f12f2242f9e-lib-modules\") pod \"perf-node-gather-daemonset-pm2kb\" (UID: \"767a2f9e-0f22-407f-8c1e-4f12f2242f9e\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-pm2kb"
Jul 31 15:43:01.229354 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:01.229284    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/767a2f9e-0f22-407f-8c1e-4f12f2242f9e-proc\") pod \"perf-node-gather-daemonset-pm2kb\" (UID: \"767a2f9e-0f22-407f-8c1e-4f12f2242f9e\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-pm2kb"
Jul 31 15:43:01.229657 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:01.229410    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/767a2f9e-0f22-407f-8c1e-4f12f2242f9e-proc\") pod \"perf-node-gather-daemonset-pm2kb\" (UID: \"767a2f9e-0f22-407f-8c1e-4f12f2242f9e\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-pm2kb"
Jul 31 15:43:01.229657 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:01.229532    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/767a2f9e-0f22-407f-8c1e-4f12f2242f9e-podres\") pod \"perf-node-gather-daemonset-pm2kb\" (UID: \"767a2f9e-0f22-407f-8c1e-4f12f2242f9e\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-pm2kb"
Jul 31 15:43:01.229657 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:01.229581    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/767a2f9e-0f22-407f-8c1e-4f12f2242f9e-sys\") pod \"perf-node-gather-daemonset-pm2kb\" (UID: \"767a2f9e-0f22-407f-8c1e-4f12f2242f9e\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-pm2kb"
Jul 31 15:43:01.230029 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:01.230003    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/767a2f9e-0f22-407f-8c1e-4f12f2242f9e-lib-modules\") pod \"perf-node-gather-daemonset-pm2kb\" (UID: \"767a2f9e-0f22-407f-8c1e-4f12f2242f9e\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-pm2kb"
Jul 31 15:43:01.241402 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:01.241373    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r79jz\" (UniqueName: \"kubernetes.io/projected/767a2f9e-0f22-407f-8c1e-4f12f2242f9e-kube-api-access-r79jz\") pod \"perf-node-gather-daemonset-pm2kb\" (UID: \"767a2f9e-0f22-407f-8c1e-4f12f2242f9e\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-pm2kb"
Jul 31 15:43:01.369984 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:01.369953    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-pm2kb"
Jul 31 15:43:01.475654 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:01.475627    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5b8776669-rgrwh_689c6bab-fc78-4cc4-81be-50463036b079/volume-data-source-validator/0.log"
Jul 31 15:43:01.544468 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:01.543788    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cxx9f/perf-node-gather-daemonset-pm2kb"]
Jul 31 15:43:01.550472 ip-10-0-142-219 kubenswrapper[2559]: W0731 15:43:01.549018    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod767a2f9e_0f22_407f_8c1e_4f12f2242f9e.slice/crio-ddc71fac84d2bb93f22c76677aaffdcb30df000688e49cf619009da8154975a8 WatchSource:0}: Error finding container ddc71fac84d2bb93f22c76677aaffdcb30df000688e49cf619009da8154975a8: Status 404 returned error can't find the container with id ddc71fac84d2bb93f22c76677aaffdcb30df000688e49cf619009da8154975a8
Jul 31 15:43:02.269369 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:02.269323    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-pm2kb" event={"ID":"767a2f9e-0f22-407f-8c1e-4f12f2242f9e","Type":"ContainerStarted","Data":"2b90447fd850ca0140dae88160bc22bb1d56b61995f8d5777526f323e6ea90ef"}
Jul 31 15:43:02.269869 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:02.269396    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-pm2kb" event={"ID":"767a2f9e-0f22-407f-8c1e-4f12f2242f9e","Type":"ContainerStarted","Data":"ddc71fac84d2bb93f22c76677aaffdcb30df000688e49cf619009da8154975a8"}
Jul 31 15:43:02.269869 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:02.269506    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-pm2kb"
Jul 31 15:43:02.289465 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:02.289404    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-pm2kb" podStartSLOduration=1.289382037 podStartE2EDuration="1.289382037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 15:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:43:02.287016054 +0000 UTC m=+3278.598573680" watchObservedRunningTime="2026-07-31 15:43:02.289382037 +0000 UTC m=+3278.600939658"
Jul 31 15:43:02.427336 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:02.427309    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2mr24_ee553c57-d835-4598-a0ee-b6fe7d685490/dns/0.log"
Jul 31 15:43:02.454974 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:02.454947    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2mr24_ee553c57-d835-4598-a0ee-b6fe7d685490/kube-rbac-proxy/0.log"
Jul 31 15:43:02.585033 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:02.584950    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mpkxl_54adc061-d1ce-4ba7-83cf-3b2968693d52/dns-node-resolver/0.log"
Jul 31 15:43:03.109335 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:03.109302    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bwcjx_16b5669e-0dde-49c0-854b-495854e99a82/node-ca/0.log"
Jul 31 15:43:04.223854 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:04.223822    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fz26d_ba20a352-de91-49e8-a2ac-b5e1b9c37ae4/serve-healthcheck-canary/0.log"
Jul 31 15:43:04.734956 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:04.734925    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dnt6w_91c153e9-2281-4901-a232-1dcd8326f94d/kube-rbac-proxy/0.log"
Jul 31 15:43:04.755977 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:04.755946    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dnt6w_91c153e9-2281-4901-a232-1dcd8326f94d/exporter/0.log"
Jul 31 15:43:04.777384 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:04.777357    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dnt6w_91c153e9-2281-4901-a232-1dcd8326f94d/extractor/0.log"
Jul 31 15:43:07.380205 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:07.380175    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-fcc9dd4db-4t79k_2cde6c4f-b5ff-4e16-a860-77959ec50852/manager/0.log"
Jul 31 15:43:07.899113 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:07.899081    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-wx8q8_1f4263e6-2f78-4867-8087-5be7e0ba9ea3/seaweedfs/0.log"
Jul 31 15:43:07.925327 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:07.925299    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-kt4jv_f79860f3-d477-422f-89f4-0b0c625f2c9f/seaweedfs-tls-custom/0.log"
Jul 31 15:43:08.289599 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:08.289511    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-pm2kb"
Jul 31 15:43:12.494565 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:12.494520    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-c6dsw_1380fa66-45b6-48b5-aee6-18f70ca49ebf/kube-storage-version-migrator-operator/1.log"
Jul 31 15:43:12.497026 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:12.496985    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-c6dsw_1380fa66-45b6-48b5-aee6-18f70ca49ebf/kube-storage-version-migrator-operator/0.log"
Jul 31 15:43:13.384640 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:13.384594    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-64wqp_88ce38fc-0695-4bfe-8292-3547c26c44d5/kube-multus/0.log"
Jul 31 15:43:13.767155 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:13.767085    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jfrf7_38a6d9ec-ffc7-4d77-b8c2-819775c1cf24/kube-multus-additional-cni-plugins/0.log"
Jul 31 15:43:13.791200 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:13.791174    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jfrf7_38a6d9ec-ffc7-4d77-b8c2-819775c1cf24/egress-router-binary-copy/0.log"
Jul 31 15:43:13.816257 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:13.816215    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jfrf7_38a6d9ec-ffc7-4d77-b8c2-819775c1cf24/cni-plugins/0.log"
Jul 31 15:43:13.841604 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:13.841573    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jfrf7_38a6d9ec-ffc7-4d77-b8c2-819775c1cf24/bond-cni-plugin/0.log"
Jul 31 15:43:13.863321 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:13.863294    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jfrf7_38a6d9ec-ffc7-4d77-b8c2-819775c1cf24/routeoverride-cni/0.log"
Jul 31 15:43:13.887459 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:13.887430    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jfrf7_38a6d9ec-ffc7-4d77-b8c2-819775c1cf24/whereabouts-cni-bincopy/0.log"
Jul 31 15:43:13.908459 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:13.908425    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jfrf7_38a6d9ec-ffc7-4d77-b8c2-819775c1cf24/whereabouts-cni/0.log"
Jul 31 15:43:13.993583 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:13.993551    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-b45r4_355d9cbb-1867-49cc-8ec2-564b4aa6daea/network-metrics-daemon/0.log"
Jul 31 15:43:14.013264 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:14.013235    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-b45r4_355d9cbb-1867-49cc-8ec2-564b4aa6daea/kube-rbac-proxy/0.log"
Jul 31 15:43:15.080121 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:15.080091    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qp8cd_c55da51d-eba3-4993-a0bb-65f32e5d73cc/ovn-controller/0.log"
Jul 31 15:43:15.098517 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:15.098477    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qp8cd_c55da51d-eba3-4993-a0bb-65f32e5d73cc/ovn-acl-logging/0.log"
Jul 31 15:43:15.127861 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:15.127832    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qp8cd_c55da51d-eba3-4993-a0bb-65f32e5d73cc/kube-rbac-proxy-node/0.log"
Jul 31 15:43:15.150804 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:15.150775    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qp8cd_c55da51d-eba3-4993-a0bb-65f32e5d73cc/kube-rbac-proxy-ovn-metrics/0.log"
Jul 31 15:43:15.173625 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:15.173555    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qp8cd_c55da51d-eba3-4993-a0bb-65f32e5d73cc/northd/0.log"
Jul 31 15:43:15.198849 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:15.198823    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qp8cd_c55da51d-eba3-4993-a0bb-65f32e5d73cc/nbdb/0.log"
Jul 31 15:43:15.222306 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:15.222276    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qp8cd_c55da51d-eba3-4993-a0bb-65f32e5d73cc/sbdb/0.log"
Jul 31 15:43:15.290020 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:15.289989    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qp8cd_c55da51d-eba3-4993-a0bb-65f32e5d73cc/ovnkube-controller/0.log"
Jul 31 15:43:16.534922 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:16.534894    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-6vfsr_567a8972-6aaf-499f-a8d8-a83da117da60/network-check-target-container/0.log"
Jul 31 15:43:17.638045 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:17.638012    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-tcn8w_a734dd47-fffc-4dfd-8946-3c3466565b8c/iptables-alerter/0.log"
Jul 31 15:43:18.310987 ip-10-0-142-219 kubenswrapper[2559]: I0731 15:43:18.310958    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-xvpvq_f6546deb-c60b-4bbb-9535-7dac6a479299/tuned/0.log"